Ben Runchey’s Integration Dev Blog

a.k.a Ben’s spot to put stuff he does not want to forget

What I Learned Yesterday 12

  • Orchestration Throttling within BizTalk – It appears that Scott Colestock has a nice approach to throttling.  Richard Seroter has it outlined on his blog here.  While I like this solution, I still wish that the good guys on the BizTalk Server dev team would provide some sort of functionality for us in this realm.  I realize it is maybe not the most important functionality, but it is probably the only feature from BizTalk 2000/2002 that I miss.
  • Archiving Pipeline Component – Ran into a situation where I wanted to archive outbound flat-files in a send pipeline.  So I looked into using the Archiving component written by Gilles Zunino a few years ago since it has worked so well for me on the inbound files.  Well after some testing it appears that there are some issues using this with the FF Assembler in the same Send Pipeline.  I tried implementing some of the solutions others have posted on their blogs, but none worked for me.  Not exactly sure what causes this, but to handle archiving my file, I used the CForwardOnlyEventingReadStream Class which is found in the Microsoft.BizTalk.Streaming namespace (and assembly of the same name).   I found a great post about using it at Mikael Hakanssons blog here about how he was using it in a component to archive messages to a TEXT field.  Basically what this class provides is the ability to have a callback for when BizTalk reads the stream to give you the opportunity at the same time.
  • Xobni Add in for Outlook – If you are using outlook you might consider trying the addin from Xobni.  It really makes “seeing across” your email intuitive.  It tracks all sorts of “social” information about your inbox. 
  • BizTalk Best Practices Analyzer – While this is not a new application, I have been trying to use it more religiously when setting up a configuration on one my client’s biztalk server group.  It just makes sure to catch things that I may miss that do not adhere to what the BizTalk team has designated as best practices.  Kind of nice to schedule running this on a regular basis (like nightly maybe) to catch when something might have gotten out of wack on your server.  It would be nice to be able to have the report be loaded into the MMC console.  Kind of a one stop shop to manage your BizTalk server. 

May 27, 2008 Posted by | BizTalk, What I Learned Yesterday | Leave a comment

What I Learned Yesterday 11

  • Google and their “exodus” – Found an article on Fortune.com earlier today about “What’s next for Google?”.  Much of the article focused on why many of the most talented people are moving on from Google.  I don’t think that those writing these articles are always in touch with what makes many technical people tick.  Working for Google was probably something that allowed many of these engineers to work on advanced projects, learning new things from people they may not otherwise have run into.  The one down side to this for these engineers is that they are working FOR other people who are making the decisions.  Many of the best engineers would ultimately like to have some sucess on their OWN terms.  This in my opinion is why many of the best people are most likely moving on to start their own thing, or have more say in what goes on.
  • Blogical sFTP adapter for BizTalk – Found a really great sFTP adapter on Codeplex.  While I still love the adapters from /n Software, if you are only going to be using the sFTP adapter, it is kind of expensive to pay the 2500/proc to buy it.  Not to mention the guys that wrote it are very quick to respond to emails (thanks Mikael!).
  • Sql Server Integration Services – I am having to work with this quite a bit these days.  I like it but there are a few things I wish it had
    • The support for Checkpoints is nice, but I wish it were improved to support multiple checkpoints within one package.  I have to basically use a custom built state machine to make this happen for myself right now.
    • Ability to have some common scripts.  I would like to have to only write common code once and share it between all of my script tasks.  I realize that I can use a .NET component to do this, but for small blocks of code I would like to be able to use a module of some sort.

May 14, 2008 Posted by | BizTalk, SSIS, What I Learned Yesterday | Leave a comment

What I Learned Yesterday 10

  • Found a great tool for exporting data from existing SQL Server Tables to a script – The SQL Scripter!  What a clever name.  You can find it at http://www.sqlscripter.com/
  • Looking into Ruby on Rails these days for a potential project and I have to say WOW!  this environment is really productive.  Just in the short time that I have spent with it I can tell it was written with how a developer needs to work as it’s main goal.  The strict MVC support, ActiveRecord for the ORM layer, and the built in support for unit testing.  While I really like it, I still really thing .NET and the CLR/DLR is the best of both worlds.  I have not had a chance to see the MVC support in the most recent version of ASP.NET (3 or 3.5, can’t remember), but my advice to Microsoft is to start building your tools to utilize your frameworks with the developer in mind.  I think that is reason number 1 why people are raving about Ruby.  The productivity in this environment is sweet!
  • What is with the script task in SQL Server Integration Services?  I would like to find out why it is not possible to just access any global variable inside of my script task!  For some reason, very smart people like to make me declare what global variables I am going to need access to before I can use them in the code.  I hope they change this in the next version!
  • I have pretty much decided to go with XSL Transformations instead of the biztalk mapper for any maps which start to get complicated for two reasons.
    • First reason is development time.  I can code XSL much faster than I can trial and error my way through using the functoids available.  Also the lack of a simple if then else type functoid can sometimes frustrate me to no end.
    • Being able to debug an XSL is easier when it is written using variable names that have context. 

April 19, 2008 Posted by | BizTalk, Ruby on Rails, T-SQL, What I Learned Yesterday | 1 Comment

What I Learned Yesterday 9

Been a while since I have apparently “learned anything” so there is much to cover

  • MSMQ is becoming more popular as Windows gets further and further into the enterprise and while I have some experience with it, I am by no means a complete expert.  This blog here does give a ton of information about MSMQ and all it entails.  While the title of this blog is “MSMQ from the Plumber’s mate”.  I think the name of the individual is John Breakwell.  I plan on reading this blog quite a bit.
  • On the topic of BizTalk Application deployment
    • The BizTalk Explorer Object model appears to provide the best intuitive access to programmatically manipulate BizTalk artifacts.  While the WMI classes are there, they are kind of archaine to work with.  This object model is pretty sweet.  Lots of good examples found in the SDK\Samples\Admin\ExplorerOM directory.
    • Automated deployment – while I was thinking about writing a script that would be a generic build – deploy – export msi type script for Biztalk, it is working out that these things become beasts onto themselves if you try to make them too generic.  Not to mention when you are giving the script to a person who really does not know that much about BizTalk, they are NOT going to be able to follow it.   So I am thinking about putting together more of a tool kit that contains a template showing how to use the BTSDeploy tool in conjunction with the Settings File Generator from Tom Abraham to ease Biztalk deployments. 
  • XSLT – Lots of stuff learned here.  I have used this sparingly in the past, but have been into it pretty good in the last few weeks.
    • Converting uppercase to lower case and vice-versa had me stumped, but found this handy article here which solved my problem. 
    • Grouping and sorting – while lots of examples show how to do this, doing it w/ multiple fields was not as easy to find.  This article here from Tech Republic shows how to get it done.  Check out the section titled “Multiple Criteria”.
    • When accessing elements in an XSL and the source document has a namespace that you have aliased with a prefix, don’t forget to use the prefix when accessing those sub elements. 
  • SaaS hosting
    • Google just released their application Google Apps the other day.  I have to day this looks pretty sweet having a whole hosted location up in the clouds to run an application from. Did I mention it is free (up to a point)!  While I think this does alot for moving forward the concept of a platform at a great price (free), and I think it will make all of those sexy social computing concepts much easier to build, it still misses some of the needs of the business apps market.  The reason I say this is the database support is based on a “Table” approach rather than a relational database approach.  While I understand why they don’t want to host a RDBMS in the sky it is still easier to write reports off these than off xml or tables. 
    • I think SalesForce is going to get crushed by this offering by Google.  How can these guys who from what I can tell are not really having much success with their platform convince people just starting out that they are worth paying for?
    • I am thinking that the tools that MS seems to be putting together are going to provide a platform based on .NET, BizTalk Services and SharePoint Services that will be a pretty good competitor to this.  Again this is going to come down to price and if MS wants to make a go of this making it free for smaller shops is definitely going to help them out.
    • Where is Yahoo in all of this?  Oh yeah they are busy telling people how great they are.  Well guys show us!

April 8, 2008 Posted by | Uncategorized | , | 1 Comment

Energy Options for the US

The topic of “energy independence” seems to always be tied to political motivations that behind that seem to be tied to isolationism.  Kind of like if we stop having to talk to the people that are causing a headache for us we will not have our headache any longer.  I never really like this bury your head in the sand mentatlity.  I think what our country should focus on are “energy options” instead.  My opinion is that over the next 15 to 20 years having fuel flexibility is what is going to provide security and economic development for the US and for the remainder of the world for that matter.  Oil, natural gas, coal, solar, wind, and the myriad of biofuels all have to be options for us.  Different fuels provide the flexibility we need to handle pollution, price fluxuations, etc. 

This article http://seekingalpha.com/article/69659-recession-i-ve-got-a-solution talks about why development of a liquified coal plant to produce jet fuel is a good idea.  He even takes it one further by pointing out how an investment in this could provide for a boost in our trade deficit.  There are many other point here also. 

March 24, 2008 Posted by | Uncategorized | Leave a comment

What I Learned Yesterday 8

  • In my past use of generics I largely used them to strongly type the parameters of a list class, or a utilitiy class.  Basically one generic type per class.  But I recently ran into a situation where I needed to use multiple generic types in one class.  I have to admit I think the coolest thing that was in the 2.0 verson of C# was the support for generics.  I love it.  here is a link to a hard core use of generics which helped me figure out how to use generics to solve my problem. http://www.itu.dk/~sestoft/gcsharp/index.html
  • Don’t short change yourself when it comes to load testing your application.  There are so many things that you cannot simulate within your app in basic unit testing and functional testing.  Certain things only appear when an app is pushed.  I really wish you people out there who think you are “Business People” would shut up and listen to the developers.  You are paying us for our experience, why don’t you listen to it for a change!  Oh, but listening is a business decision, not something we want to leave up to the developers!

March 12, 2008 Posted by | C# Generics, What I Learned Yesterday | Leave a comment

What I Learned Yesterday 7

  • It has been a while since I have had to really go at things using T-SQL.  Lots of my recent experiences have been around CRUD type operations where we would read the data into an object model work on it and then save it back to the database.  But I am working on a project where we are loading a temp table and then inserting/updating records into another table.  To get the identity columns of the newly inserted or newly updated records the OUTPUT keyword in T-SQL is a life saver.   This is only available in SQL Server 2005/2008 though.  This link here is a good overview of what it can do.
  • I spoke of how cool the use of Over is with rownumber, but here is an article where the guy is improving on top.  I have to admit the things that are in SQL 2005 and newer from just a T-SQL standpoint are pretty sweet.  They make writing some of the useless looping logic that permiated all aspects of my “business logic” a much easier task. Shows how long I have been doing XML and Integration that I have not spent any large amounts of time learning these new features (which are not so new at the time I am writing this) http://www.sqlmag.com/Articles/ArticleID/49240/49240.html?Ad=1
  • Hiding the Results Pane (or should I say pain?) in the SQL 2005 Management Studio is something I have wanted for quite some time.  I have always liked being able to have a full screen for editing my T-SQL script.  I have finally figured out how to make this happen thanks to this guy here.  http://www.jjerome.com/projects/2007/04/showhide-results-pane-sql-server-2005.html
  • To output a null date to a FlatFile Schema in Biztalk set the format of the date field to yyyyMMdd.  Then use the pad character to zero fill if you need. Beautiful

March 7, 2008 Posted by | BizTalk, T-SQL, What I Learned Yesterday | Leave a comment

What I Learned Yesterday 6

  • To enable edit and contiue I had to uninstall the Windows SDK debugging tools.  Apparently this had a profiler running which did not allow for me to do the edit and continue in vs 2005.   Life is much better now.  I had actually never used this in 2005 previously.
  • BizTalk CrossReferencing API – looks like this is the way to solve those pesky file counters that some people like to use to avoid duplicate files or as a reference point. Again since I feel like all I do is regurgitate things that other people have already writted good blog posts on I will just do some simple link forwarding on this one.  The first one is by a guy named Eric (not sure about his last name) and can be found here.  While the other one is by Brian Loesgen and can be found here.   
  • Using the Looping functoid and the Table looping functoid together does not always work so well.  Please see my sample post to follow shortly on why these two don’t work together.
  • BizBert has put together a great group of articles about the BizTalk Mapper.  Lots of information about functoids, performance, etc. You can find it here.

February 28, 2008 Posted by | BizTalk, Visual Studio 2005, What I Learned Yesterday | , , | Leave a comment

Finding out the mapper can do what I want it to do

This is a follow up post to my previous post talking about how I used some XSLT with an interim schema to get some mapping done.  In my travels around the internet since then I have found a post from Jeff Lynch about how to use the mapper to map “Small Loops“.  Well this was exactly what I was trying to get done in my previous post.  Simply put the looping functoid can get done what I need.  No more interim schema!

 here is the new and improved map
SelectiveMappingSmallLoops

Notice how I linked the PONum, ShippingDate attributes to the Looping functoid seperately from the Item element.  Once I slowed down long enough to think about what the XSLT this would generate it made sense.  Each element or attribute plugged into the left side of the Looping functoid will have it’s own loop created for it.  That is the key here.   I have also updated the source code for this guy.

 I have to say, I like this solution much better!

February 19, 2008 Posted by | BizTalk | | 1 Comment

Some XSLT for when you just can’t get the mapper to do what you want it to do

Update – Please see my subequent post titled “Finding out the mapper can do what I want it to do” which makes this “hack” not necessary any longer 

Recently I have been spending some (maybe too much) time on the msdn forums to see if I can help out anyone that is having a problem with BizTalk.  I figure since most of my knowledge comes from reading blog posts, I need to do something that gives back to the world. 

So on friday a post showed up about needing to map from one format with data in a non repeating header and moving it to a repeating element in a destination document.  You can read actual post here.  An example of the source and destination would be as follows

The source schema looks like:

SelectiveMappingSourceDocument

 And our destination schema looks like

SelectiveMappingDestinationDocument

So as you can see we need to take the PONum and ShippingDate attributes from the Header element of the source document and put them into two different MainLoop elements in our destination document.  Then take the repeating Item elements from the source and fill out the remaining MainLoop elements in the destination document. 

Getting the PONum and ShippingDate out proved to be a bit more challenging than I had orignally thought.   I did not want to write just straight XSLT and by pass the mapper.  That is always an option, but then I am not really using the tools available.    What I decided to do was use inline XSLT in a scripting functoid to copy the Source\Header\PONum and ShippingDate fields to the MainLoop.  So I came up first with a map that looked like this

 SelectiveMappingFirstAttempt

But the Xml was not what I wanted… (not even close!)

SelectiveMappingFirstAttemptResult

So when looking at the xslt the map was creating I realized that  getting a MainLoop element for both my PONum and ShippingDate field would require me to have an “empty” spot in my destination schema that I could push the XSLT from my two scripting functoids into.  So I created a temporary destination schema that contained a “PlaceHolder” for me to overwrite.

SelectiveMappingModifiedDestinationSchema

Now my map looks like this…

SelectiveMappingCorrectMap

Which produces the correct output…

SelectiveMappingDestinationDocument

Now I know, if this is a huge document, or I am doing lots of these throughout the day and performance is a very big consideration the straight XSLT approach would be more performant, but until I need to have the speed, I will take the maintainability.  You can download the project that demonstrates this here

 Mape1082 I hope this helps you somehow. 

 

February 19, 2008 Posted by | BizTalk | 1 Comment