Joe Cincotta: Thoughts and such…

Icon

Nerdism for the masses.

BizBlox Find/Fetch methods explained

by Stephen Trembath

Just a quick summary of the DOCollection methods and how they work at the query level.

I will add to documentation eventually once I refine this a bit more, and maybe reduce the number of methods:

Take for example a CustomerCollection, comprised of Customer DataObjects. For the below examples the query will find customers with FirstName=Stephen

SetCurrentQuery
Simply sets a SqlBuilder as the query to run on the DOCollection – does not actually run any SQL.
To run the query you have to use one of the below methods. If you do not run any of the below methods, and try to access DataObjects within the collection, it will default to running the Find() method.

Find
For the current query, selects all IDs:
e.g. SELECT CustomerID FROM Customer WHERE FirstName=’Stephen’

Only when the actual Customer objects are accessed is the DataObject loaded via a new SQL query:
e.g. if you do custColl(0) it will run: SELECT Customer.* FROM Customer WHERE CustomerID = ‘……’

Iterating through a large collection, can result in a lot of queries – i.e. one to select all IDs, and one to select details of each DataObject you access.

Fetch
Retrieves all data for all matching rows in the query:
e.g. SELECT Customer.* FROM Customer WHERE FirstName=’Stephen’

This returns more data, but there is no need for follow-up queries to retrieve customer information.

FindAll
Works like a Find, but disregards the current query, and instead finds all matching IDs within the table
e.g. SELECT Customer.CustomerID FROM Customer

Should be used sparingly.

FetchAll
Works like a Fetch, but disregards the current query, and instead finds all matching IDs within the table
e.g. SELECT Customer.* FROM Customer

Should be used sparingly.

Count
If the DOCollection has been populated by a Find/Fetch etc, this will just return the number of rows from the collection itself.

If the DOCOllection has not been populated, this will do a SELECT Count(*) FROM Customer

FindRange
Returns a subset of the rows retrieved by the query. e.g. custColl.FindRange(50, 99) will skip the first 50 rows and return the following 50.

It does this by finding all IDs that match the query, while using a SELECT TOP to minimize the upper range

(e.g. SELECT TOP 100 CustomerID FROM Customer WHERE FirstName=’Stephen’)

It then manually removes in code the IDs that are not needed, e.g. the first 50 IDs in this case.

After that, accessing each DataObject will result in a SELECT Customer.* FROM Customer WHERE CustomerID = ‘……’ type query.

FetchRange
Similar to FindRange() except that it does a SELECT(*) to retrieve all data, e.g. custColl.FetchRange(50, 99) will skip the first 50 rows and return the following 50.

(e.g. SELECT TOP 100 Customer.* FROM Customer WHERE FirstName=’Stephen’)

It then manually removes in code the DataObjects that are not needed, e.g. the first 50 DataObjects in this case.

Generally not recommended.

FetchSelected
This is a new method that is still under development for general use, and needs to be defined better. It is best used where performance is crucial, and where the query is not overly complex.

It is similar to FindRange but uses SQL Server 2005′s built-in paging for efficiency, using the ROW_NUMBER function:
http://www.davidhayden.com/blog/dave/archive/2005/12/30/2652.aspx

It also only selects the columns explicitly entered in the SqlBuilder (you can still add a SELECT *)

However, with that comes a greater chance for limitations in queries, and should only be used for now where fullt tested.

The ROW_NUMBER function will only return from the database the actual rows that are required, and there is no need for manual removal in code.

e.g. Case 1:
Dim mySql as New SqlBuilder(“Customer”)
mySql.addSelect(Customer.

Columns.CustomerID)
mySql.addSelect(Customer.Columns.FirstName)
mySql.addSelect(Customer.Columns.LastName)
mysql.addWhere(Customer.Columns.FirstName, ‘Stephen’)

custColl.FetchSelected(50, 99)

The query will be similar to the following, but using the ROW_NUMBER syntax to retrieve the desired rows:
SELECT Customer.CustomerID, Customer.FirstName, Customer.LastName FROM Customer WHERE Customer.FirstName=’Stephen’

e.g. Case 2:
Dim mySql as New SqlBuilder(“Customer”)
mySql.addSelectAll()
mysql.addWhere(Customer.Columns.FirstName, ‘Stephen’)

custColl.FetchSelected(50, 99)

The query will be similar to the following, but using the ROW_NUMBER syntax:
SELECT * FROM Customer WHERE Customer.FirstName=’Stephen’

Filed under: agile development, BizBlox, Open Source, Software Development

SaaS Mentor can help…

Last week I started SaaS Mentor consulting – a business designed to help other business owners succeed.

How?

I have been fortunate enough to work with the owners of various sized businesses over the past decade – I found that nearly all of these businesses faced similar problems; problems around effective sales process, marketing, time keeping and project management. The other major issue for them was the cost of IT and the people required to maintain it all. This cost generally hit those businesses at the worst possible time and it was not usually a small cost either…

The thing I noticed was that over the past year or so there has been an influx of really powerful business tools available on the internet which are provided as subscription services. This Software As A Service model is specifically suited to small-to-medium sized businesses since it reduces the need for in house servers and IT staff costs to maintain them. No big investments and great support.

The idea for SaaS Mentor came about when I started working on a simple approach to consulting for small-to-medium sized business which used several Software As A Service packages as the tools to provide a unique solution for a business problem. This consulting gave big results quickly and made the process of transformation easy.

So, If you know a business owner who may need some help, I would ask that you forward the link to this page on to them and let them know what we’re doing. If you feel like your business could benefit from this kind of strategy or that something in your business is not quite right, I would love to have a chat with you and see if what we’re doing could work for your business. Just email me: joe@saasmentor.com.au

Filed under: agile development, Industry Opinion, pixolut

The Difference Between Good and Great Developers

This is a review which Terry – one of the dev guys here at Pixolüt – wrote which I thought was just great. This shows how important it is to have a good handle on many skills as a developer. Its no good to just be great at C# or Java – a great developer can get across the entire cross-section of the enterprise application and find the right place to expend effort in optimizing the system.
 
“We had a win this week, dramatically reducing the length of one of our overnight processes by adding a single index to one table in SQL Server.
 
Our customers had complained that the nightly outage for refreshing data had shifted so late that their screens were out of action around 8am. This was just when they most want them available, to catch the eyes of early morning commuters.
 
Checking the System Status messages showed that the import of data was completing shortly after 1am, but the rendering of that data was taking over 6 hours. A further check of log files for the rendering process revealed a likely problem area, a single function consistently taking one to one and a half minutes. Running with breakpoints in development let us drill down to 3 database calls that might be candidates for improvement.
 
In SQL Server, there’s a feature that displays the estimated execution plan for a query. Using this on the 3 calls and checking the “Estimated Subtree Cost” showed 0.006 for one, 0.01 for the next and 9.4 for the third, obviously the problem was here.
 
The problem query retrieves rows from a large table via a foreign key that was not part of any index. Adding a secondary index on that column reduced the estimated cost from 9.4 to 0.09.
 
The night after adding the index, the render improved from just over 6 hours to just under 2, the entire nightly refresh process completing well before 4am.”

Filed under: agile development, pixolut, Software Development

Golden Rules of Source Control

There are some important things to remember which can serve to make addition of developers to the team simple and make it easy for merging and tracking development streams. The flipside of this is that not following these simple rules can muddy up source control and turn something simple in to something complicated. Here’s some golden rules.

If it is an automated output – it should not be there
IDE’s and build scripts can often generate intermediate or temporary files. You as a developer should know the difference between throwaway files like the ‘.user’ files which Visual Studio creates and the project files which are essential. Face it, there will always be a merge problem between developers when it comes to project files, why make it worse?

If it is copied from somewhere – keep it in one place at the source
There should only ever be one copy of a file. Multiple versions is a big NO NO! If, for example, you have references to a set of DLL files for a few projects in their BIN folders, create a DLL folder up at the solution level and make each project refer to that folder as the source for the ‘hintpath’. By just having copies of DLLs in every bin folder of a solution, you create DLL HELL as versions of libraries change and each developer in the team is at a different stage of development.

If it is crap – flush it, If you see it should not be there – remove it
Make a point of never committing a ‘.bak’ file or any other crazy file which really should not be there, as a developer your job is to introduce order, not chaos. Spend an extra 10 seconds when you are committing to make sure junk is not there. If you are working on a merge or some other task and you notice something should not be there, it is your duty as a citizen of the project environment to clean it up.

Personal configuration does not belong in a repository
Configuration files will always be a point of conflict as they need to be stored for posterity, but not much more. Make a point of only ever committing a ‘golden’ config which acts like a dictionary of ‘config keys’ with example values. The real thing should not be in source control unless you are keeping it as a backup for a server – in which case it belongs OUT of a solution and in a seperate place specifically for that purpose.

Filed under: agile development, Software Development

Un-SVN … for those times you just need to start again!

Download UnSVN which is a little Win32 command line tool which will search for and delete all the hidden .svn or _svn folders in a folder so that you can ‘start again’ with your working copy and commit as if it were new.

I have found that a lot of the time we do a BIG merge and are really starting a whole new repository and would like to just blow away the SVN directories from my working copy so that I can just do a fresh check out and commit process to a new trunk or branch.

Its not that sexy so it will try to just do a local delete of .svn if you do not supply any parameters. Instead, try using -help to get a list of options. Not bad for 10 minutes work…

Example:
unsvn -r -u -t c:\developer\newProject

[Note: there was a crazy problem last night where I found that directories and files which were under the .svn folder which were marked as read only would not delete. This was fixed by just changing folder permissions - I found the solution here.]

UPDATED 4/3/2010! I have changed the link in this article to point to VERSION 2 of this great little utility. It traverses large filesystems recursively and works a treat. Download UnSVN version 2.0 to fix your SVN woes! Requires .Net framework 2.0 to be installed.

Filed under: agile development, Open Source

Getting Agile with BaseCamp

[37 Signals recently updated some great features on BaseCamp which fundamentally affect how we do Agile with our team, so I thought I would update this article as it still seems to get a lot of traffic.] 

After a lot of agony, I have found that there are really two ways of approaching a small lightweight agile development team with management tools. Rigid structure tools which try to implement a flavour of an agile methodology or loose structure tools which allow for your own approach to agility. There are actually plusses and minuses against each approach; so this statement is not about taking the steam out of Version One or Tackle by any means, but it is about our journey of discovery in dealing with non-technical customers and LOTS of disparate development tasks using only a thin veneer of project management on top.

We spent a lot of time looking at PMM tools like ProjectNet (which has a SCRUM module) and also dedicated agile management tools. We used Version One  for some time and whilst it was a JOY to use it really was focussed on a small area of scrum whilst overall project management was still left hanging. We just didn’t have enough management resource to deal with this upkeep of a micromanagement approach as well as all the project based high level demands of customers. We were also growing a lot (people and client wise) and V1 charge by user seat and it aint cheap for a company like mine – so with some regret we moved…

I tried in earnest to use Tackle, an open source alternative which is nice, but pretty immature. It shows in its lack of usability and it is still trying to enforce these approaches to Agile which we felt were hard to deal with.

After a brief attempt at using ProjectNet – which was inordinately complex to configure – we settled on BaseCamp. Its fees are not based on ‘per seat’ so the bigger you get the more cost effective it becomes. Its also very very flexible. Its not trying to impose a methodology on you, instead its got a very freeform approach to the architecture of a PMM tool and allows you to create the framework and the procedural rules to an extent.

Below is an introduction to how we use BaseCamp as a parrallel management tool for dealing with high level project mangement needs whilst also giving our developers the toosl they need to manage burn downs and stay agile…

1. Using MESSAGES for briefs (stories), bugs and support requests:
A brief for functionality, a bug request or a support ticket should be a MESSAGE entered in to BaseCamp with the appropriate category assigned to it. Once the brief is processed or completed we can change its state to completed by changing the category under which the item is filed. Just make sure that Admins set the MESSAGE status when you complete work. (ie: you need to have project managers or account directors or scrum masters with ADMIN access)

2. Using TODO Lists:
Each MILESTONE can have a TODO LIST. This TODO LIST should be set up to allow you to track time against it. By having a single goal with trackable time you can then start making the actual TODO ITEMS. Think of the TODO ITEMS for a goal as being like BurnDown chart for a scrum-oriented team goal. This allows an individual or mini team to manage their own development specific goals without making the high level project plan dirty.

This approach to using TODO LISTS also means that you can track your TODO LIST progress against a single Milestone which means no ‘Punting’ or moving all your tasks from week-to-week. Milestones can be set up like a sprint yet they can move if other demands come in without having to micromanage a shopping list of items you thought you could get done this week and now can’t.

3. Using MILESTONES:
MILESTONES give us the ability to create project goals and timings for our clients. Generally an Admin (Project Manager / Account Manager / Scrum Master) will create the MILESTONE after conference with developers regarding time and availability subject to customer priorities.

MILESTONES are the glue which bind together User Stories (MESSAGES) and Burn-Downs (TODO LISTS); once you create a MILESTONE for a User Story you can now bind the written brief (MESSAGE) to it. The TODO LIST allows the same function of selecting a MILESTONE and therefore viewing a MILESTONE will give the user visibility in to both the customer facing aspect and the developer facing aspect.

Depending on the technical ability of the customer, they can browse the TODO burndown or stay at the high level of MILESTONES and MESSAGES. They are connected yet (as mentioned above) they do not interfere with a simple high level understanding of a project.

Filed under: agile development, Software Development

We’re hiring

Click here to see more or apply: mycareer.com.au/5098411

Full Time .Net Developer :
Get on the Cutting Edge!

COMPANY PROFILE
Pixolüt Industries is a research and development company which incubates start-up companies to develop their ideas in to successful businesses. We also deal with mature businesses looking for technology partners when they want to expand in to new markets.

Position Overview
We need a very bright and motivated developer for a full time role. You get to work on a broad spectrum of projects using everything from web services to AJAX and the new WPF in .Net 3!

Work on the cutting edge of the Microsoft technologies, get time to experiment and research as well as work on several amazing systems for local and global clients.

We actively support open source projects and encourage contribution to the community from our ‘labs’ projects which are the maturing of some of our research and development.

You will come in to existing projects and work with a team of excellent developers and you would be lead by Joe Cincotta; which will provide you some great mentoring and learning opportunities. You would also get training on the job with these bleeding edge technologies and the skills and processes to back them up. We really want people who can learn as an ongoing process – we don’t expect people with loads of .Net 3.0 experience!

Working at Pixolüt you will be in a very casual environment in our lab located in Sydney’s south. Defninitely no suits or ties necessary. Its near the beach and we encourage a life/work balance – so you can take time to enjoy it… 

There is no peak hour traffic ever, its just a short walk to the train station and there is free all day parking too. Perfect! On average its a 30 minute commute from the city.

MAP LINK:
http://preview.tinyurl.com/2tm64d

SUBMISSION GUIDELINES
Please provide a concise resume in .DOC or .PDF format. Ensure your resume has evidence of education history and skills.

To be eligible to apply for this position you must have an appropriate Australian or New Zealand work visa.

Click here to see more or apply: mycareer.com.au/5098411

Filed under: agile development, Open Source, pixolut, Software Development

VersionOne Agile Development Platform

I need to give a blatant plug to VersionOne for producing a fantastic Agile development platform. We are in the process of implementing SCRUM within our development teams for all projects and its just fantastic for managing the development iteration process and also tracking features and bugs.

Why the plug? Its not just the software – its the entire approach – its free for teams up to 5 users. Even the enterprise version is reasonable priced and you can deploy it how you want. Its really just a great company with a great product and I am ecstatic I stumbled on them.

I would like to actually integrate our existing defect tracker (TestTrack Pro from Seapine Software – another fantastic product) in to VersionOne – which I know can be done… its just a matter of time to do the integration which I don’t have. We have actually configured all our software to provide live crash reporting directly in to TestTrack so I am reluctant to leave it.

Well done guys…

Filed under: agile development, pixolut

A Practical Exmaple of Multi-Threading

I will briefly describe the way we improved performance using multithreading below, but first – please understand I was actually pleased that DevX are willing to do the ‘Gauntlet’ series on Intel home turf (their dev center on DevX) so the subterfuge was really referring to the general theme of Intel advertising aimed at developers lately and not this specific series. Please keep binging it on!

One thing you should realize though is that I honestly don’t think huge numbers of organizations are looking to optimize their code for multi-core architectures just yet – however the big selling point for many organizations right now is the immediate ROI for organizations in power savings both in computational efficiency per watt and also in reduced cooling and associated infrastructure costs when moving to multi-core. Its real.

Ok, iVisual performance improvements: let me explain a very brief overview of how we segment the system from a data-model perspective to help understand how we break down the threading model.

1: We have Customers
2: Customers have multiple Locations
3: Locations have multiple Displays

The thing which takes the time is the render pipeline and is the focus of this examination. There are three stages to the render pipeline:
1: The first stage downloads XML feeds from 3rd party data sources and homogenizes the data into a database
2: The second stage composites the data on to the customer-location templates and downloads third party images referred to in the data feeds
3: The system packages each location in to a compressed package for distribution

When we started in a small scale proof of concept the whole thing was sequential and as it scaled it took hours. Multithreading was the answer, but it needed a clear strategy and this is how we broke it down:

First we looked at the stages of the render pipeline and examined the code to look for immediate code optimization (not necessarily multithreading, just common sense performance improvements). Now, I don’t have a heavyweight profiler but I was still able to analyze my code without fancy tools – the answer is in two things, use Windows perfmon to watch relationships between CPU, Disk, Network and Database when running strenuous code and secondly, just by using Log4Net (The Apache open source logging project) and taking a DateTime.Now before and after a block of code which you suspect of being ‘hot’ you can easily see performance metrics and you optimize the code.

Once the optimization was done we look at where we can see a use for multithreading. The first download stage was really quite linear and the process of downloading a large chunk of data was the thing which took the most time, so we left it alone.

The second stage clearly could do with multi-threading, but how should we split it out. First, we changed the way images get downloaded – we started caching them as we parsed each template; but we used a common directory for storing files. The image caching engine would go and download an image when it doesn’t have it – so it would block the caller until it came. The ad-hoc nature of the process meant that we should really rearchitect the whole approach.

If we sequentially pre-cached every image before rendering we would know we had every image, but due to the blocking nature of the web calls to retrieve images – it would still take forever. We put the process of downloading an image out to a thread – but don’t just loop through every image and run the process of downloading to a thread as this will saturate your network and CPU and OS and nobody will want to play nice with you anymore. We use an ‘artificial’ thread-pool – that is our own round-robin model pool of thread where we control how many are running at once. This means we needed some fault tolerance in case of thread collissions where multiple locations for the same customer use the same image and we happen to try and download it at the same time. Its highly improbable, but as an aside, this is a good example of the sheer complexity which multihtreading adds to system design.

We also chose to set the parameters for determining the size of the thread pool as an application configuration parameter which means we can tweak performance of the system without recompiling.

If you’re really clever you will see how we chose to handle our logical breakdown of the system to suit multithreading. We used Locations as the target for the main threading model – so instead of looping through a list of locations and calling Location(i).Render(); we create a group of worker classes which contain references to each Location object and when run will Render() their location. Again, we use the artificial thread-pool design to manage how many locations are being rendered concurrently. Because a location deals with its own content in its own directory structure, we didn’t have to worry about any thread contention or race issues with rendering.

Finally, the CPU intensive process of compression was also moved to a thread-pool model and again it improved performance dramatically.

The change to multiple thread-pools running meant we needed to have an orchestrator which managed the seperate thread pools and ensured that only one was running at once and that they could not be invoked multiple times (through the web administration interface) so we ended up with a ‘Render Pipeline’ which was the only way to invoke on these three elements for added safety.

The really funny thing is that the most CPU intensive part of the process, the compression stage, is the fastest now – we use a quad-xeon with HT setup and it makes short work of these things.

The place where mutithreading really came to shine is in allowing our code to improve the overall load across the system – not just raw CPU improvements. We improved our caching speed by 10 times because our network was able to handle 10 concurrent connections to the content servers for precaching images.

Multithreading also allowed us to push the database server a little more during the rendering process which is both CPU intensive and massively IO intensive.

Bottom line: Multi-threading on Multi-core SMP architectures allows your code to get more than just extra CPU cycles – you can squeeze more power out of your whole system by removing latency and unnecessary waiting.

Hope this helps shine some light on a practical implementation of threading.

Filed under: agile development, Software Development

No Green Lights For You!

Argh, I am seeing nUnit in my sleep. No green bars for you! Only red! Nooooohooohohohoo! Ah well, nothing this complicated should be easy to debug. Did I mention iTOK is about to hit V6! I can’t go in to details suffice to say that its a whole new beast and it should be on the streets soon.

Filed under: agile development, Software Development

Follow

Get every new post delivered to your Inbox.