Joe Cincotta: Thoughts and such…

Icon

Nerdism for the masses.

BizBlox Find/Fetch methods explained

by Stephen Trembath

Just a quick summary of the DOCollection methods and how they work at the query level.

I will add to documentation eventually once I refine this a bit more, and maybe reduce the number of methods:

Take for example a CustomerCollection, comprised of Customer DataObjects. For the below examples the query will find customers with FirstName=Stephen

SetCurrentQuery
Simply sets a SqlBuilder as the query to run on the DOCollection – does not actually run any SQL.
To run the query you have to use one of the below methods. If you do not run any of the below methods, and try to access DataObjects within the collection, it will default to running the Find() method.

Find
For the current query, selects all IDs:
e.g. SELECT CustomerID FROM Customer WHERE FirstName=’Stephen’

Only when the actual Customer objects are accessed is the DataObject loaded via a new SQL query:
e.g. if you do custColl(0) it will run: SELECT Customer.* FROM Customer WHERE CustomerID = ‘……’

Iterating through a large collection, can result in a lot of queries – i.e. one to select all IDs, and one to select details of each DataObject you access.

Fetch
Retrieves all data for all matching rows in the query:
e.g. SELECT Customer.* FROM Customer WHERE FirstName=’Stephen’

This returns more data, but there is no need for follow-up queries to retrieve customer information.

FindAll
Works like a Find, but disregards the current query, and instead finds all matching IDs within the table
e.g. SELECT Customer.CustomerID FROM Customer

Should be used sparingly.

FetchAll
Works like a Fetch, but disregards the current query, and instead finds all matching IDs within the table
e.g. SELECT Customer.* FROM Customer

Should be used sparingly.

Count
If the DOCollection has been populated by a Find/Fetch etc, this will just return the number of rows from the collection itself.

If the DOCOllection has not been populated, this will do a SELECT Count(*) FROM Customer

FindRange
Returns a subset of the rows retrieved by the query. e.g. custColl.FindRange(50, 99) will skip the first 50 rows and return the following 50.

It does this by finding all IDs that match the query, while using a SELECT TOP to minimize the upper range

(e.g. SELECT TOP 100 CustomerID FROM Customer WHERE FirstName=’Stephen’)

It then manually removes in code the IDs that are not needed, e.g. the first 50 IDs in this case.

After that, accessing each DataObject will result in a SELECT Customer.* FROM Customer WHERE CustomerID = ‘……’ type query.

FetchRange
Similar to FindRange() except that it does a SELECT(*) to retrieve all data, e.g. custColl.FetchRange(50, 99) will skip the first 50 rows and return the following 50.

(e.g. SELECT TOP 100 Customer.* FROM Customer WHERE FirstName=’Stephen’)

It then manually removes in code the DataObjects that are not needed, e.g. the first 50 DataObjects in this case.

Generally not recommended.

FetchSelected
This is a new method that is still under development for general use, and needs to be defined better. It is best used where performance is crucial, and where the query is not overly complex.

It is similar to FindRange but uses SQL Server 2005′s built-in paging for efficiency, using the ROW_NUMBER function:
http://www.davidhayden.com/blog/dave/archive/2005/12/30/2652.aspx

It also only selects the columns explicitly entered in the SqlBuilder (you can still add a SELECT *)

However, with that comes a greater chance for limitations in queries, and should only be used for now where fullt tested.

The ROW_NUMBER function will only return from the database the actual rows that are required, and there is no need for manual removal in code.

e.g. Case 1:
Dim mySql as New SqlBuilder(“Customer”)
mySql.addSelect(Customer.

Columns.CustomerID)
mySql.addSelect(Customer.Columns.FirstName)
mySql.addSelect(Customer.Columns.LastName)
mysql.addWhere(Customer.Columns.FirstName, ‘Stephen’)

custColl.FetchSelected(50, 99)

The query will be similar to the following, but using the ROW_NUMBER syntax to retrieve the desired rows:
SELECT Customer.CustomerID, Customer.FirstName, Customer.LastName FROM Customer WHERE Customer.FirstName=’Stephen’

e.g. Case 2:
Dim mySql as New SqlBuilder(“Customer”)
mySql.addSelectAll()
mysql.addWhere(Customer.Columns.FirstName, ‘Stephen’)

custColl.FetchSelected(50, 99)

The query will be similar to the following, but using the ROW_NUMBER syntax:
SELECT * FROM Customer WHERE Customer.FirstName=’Stephen’

Filed under: agile development, BizBlox, Open Source, Software Development

We’re Hiring: .Net Web Developer

Spiral Digital is the most prolific Facebook applications developer in Asia Pacific.

We are looking for a talented Developer to work with a fantastic growing team and develop their skills in social media using .Net techologies whilst working with global brands and agencies in Australia and across Asia.

The Developer’s primary responsibility is to work with the Designer and Producer with direction and mentorship from the Director of Technology writing code solutions against specifications supplied supplied by the Producer.

On top of regular development work, you get time to experiment and research as well as contribute to open source projects.

You will be working with a team of other excellent developers and you will be provided mentoring opportunities and training as well as learning industry best practices.

We really want people who want to learn as an ongoing process and love to write code – we don’t expect you to have huge amounts of Facebook experience, we would rather you just be yourself, learn as you go and be part of our team.

Working at Spiral Digital you will be in a very casual environment in our lab located in Sydney’s south. Definitely no suits or ties necessary. Its near the beach and we encourage a life/work balance – so you can take time to enjoy it… There is no peak hour traffic ever, its just a short walk to the train station and there is free all day parking too. On average its a 30 minute commute from the city.

SUBMISSION GUIDELINES Please provide a concise resume in .DOC or .PDF format. Ensure your resume has evidence of education history and skills. To be eligible to apply for this position you must have an appropriate Australian or New Zealand work visa.

Applications should be sent to jobs@pixolut.com

Filed under: Jobs, pixolut, Software Development

BizBlox 1.7.2 Release

The latest update to BizBlox has been released today. This update has fixes for unicode support and a small issue with transaction management.

  • BizBlox used to use the local database collation settings for handling content, however this update forces unicode coalescence to comply with the way .Net handles strings (all unicode!) and makes dealing with multilingual content a no-brainer.
  • Failed transactions will no longer throw ‘null reference exception’.

Get the latest release from http://www.pixolut.com/bizblox

Filed under: BizBlox, Open Source, Software Development

Why we moved to Google Code

We have been using SourceForge for a long, long time hosting all our Open Source projects. It was a tough decision to move away from SourceForge, however as our teams and projects grew – and as time went on – we felt SourceForge had lost touch with some of the fundamentals of software development for the sake of monetization.

Over 2008 I saw SourceForge move towards a services based model to attempt to support the projects which reside on it. In and of itself this is a great idea, however the real problem was that the world of web based applications had rocketed ahead whilst the core platform of SourceForge felt like it lagged behind.

Google Code was launched way back in August 2006 and has adopted the typical Google approach to its developer platform. This approach means that adding content, code and downloads is super simple and whilst the Issue Tracker in SourceForge was turned off for all our projects since it was so cumbersome, the Google Code Issue Tracker is just a joy to work with.

Overall, we feel that the impact on development and team collaboration will be vastly improved by making the move.

The Pixolüt Industries projects on Google Code are:

BizBlox

PreNIS

xReplace

Filed under: Google, Industry Opinion, Open Source, Software Development

The Difference Between Good and Great Developers

This is a review which Terry – one of the dev guys here at Pixolüt – wrote which I thought was just great. This shows how important it is to have a good handle on many skills as a developer. Its no good to just be great at C# or Java – a great developer can get across the entire cross-section of the enterprise application and find the right place to expend effort in optimizing the system.
 
“We had a win this week, dramatically reducing the length of one of our overnight processes by adding a single index to one table in SQL Server.
 
Our customers had complained that the nightly outage for refreshing data had shifted so late that their screens were out of action around 8am. This was just when they most want them available, to catch the eyes of early morning commuters.
 
Checking the System Status messages showed that the import of data was completing shortly after 1am, but the rendering of that data was taking over 6 hours. A further check of log files for the rendering process revealed a likely problem area, a single function consistently taking one to one and a half minutes. Running with breakpoints in development let us drill down to 3 database calls that might be candidates for improvement.
 
In SQL Server, there’s a feature that displays the estimated execution plan for a query. Using this on the 3 calls and checking the “Estimated Subtree Cost” showed 0.006 for one, 0.01 for the next and 9.4 for the third, obviously the problem was here.
 
The problem query retrieves rows from a large table via a foreign key that was not part of any index. Adding a secondary index on that column reduced the estimated cost from 9.4 to 0.09.
 
The night after adding the index, the render improved from just over 6 hours to just under 2, the entire nightly refresh process completing well before 4am.”

Filed under: agile development, pixolut, Software Development

Golden Rules of Source Control

There are some important things to remember which can serve to make addition of developers to the team simple and make it easy for merging and tracking development streams. The flipside of this is that not following these simple rules can muddy up source control and turn something simple in to something complicated. Here’s some golden rules.

If it is an automated output – it should not be there
IDE’s and build scripts can often generate intermediate or temporary files. You as a developer should know the difference between throwaway files like the ‘.user’ files which Visual Studio creates and the project files which are essential. Face it, there will always be a merge problem between developers when it comes to project files, why make it worse?

If it is copied from somewhere – keep it in one place at the source
There should only ever be one copy of a file. Multiple versions is a big NO NO! If, for example, you have references to a set of DLL files for a few projects in their BIN folders, create a DLL folder up at the solution level and make each project refer to that folder as the source for the ‘hintpath’. By just having copies of DLLs in every bin folder of a solution, you create DLL HELL as versions of libraries change and each developer in the team is at a different stage of development.

If it is crap – flush it, If you see it should not be there – remove it
Make a point of never committing a ‘.bak’ file or any other crazy file which really should not be there, as a developer your job is to introduce order, not chaos. Spend an extra 10 seconds when you are committing to make sure junk is not there. If you are working on a merge or some other task and you notice something should not be there, it is your duty as a citizen of the project environment to clean it up.

Personal configuration does not belong in a repository
Configuration files will always be a point of conflict as they need to be stored for posterity, but not much more. Make a point of only ever committing a ‘golden’ config which acts like a dictionary of ‘config keys’ with example values. The real thing should not be in source control unless you are keeping it as a backup for a server – in which case it belongs OUT of a solution and in a seperate place specifically for that purpose.

Filed under: agile development, Software Development

Essential nAnt Utilities from Pixolüt

Download the nAnt Utilities from here

This is a set of nAnt includes which we use in our build pipeline which I thought may be helpful to other developers using .Net and nAnt… Download them and give them a shot…

Whats included:

build.include handles .Net compilation of projects of any size and also provides build output and error management. This also allows for multiple .Net platforms to be targetted from a single nAnt script, which is something nAnt does not do natively.

ftp.include handles automated FTP of files to a server from within nAnt

prenis.include processes a Nullsoft Scriptable Install System (NSI) file using the PreNIS pre-processor.

svn.include will do most automated gets and checkouts including automated version labels on tags using SubVersion. This is a very useful incude since the current Subversion support in nAnt is rather poor.

How to use them:

These properties are used for all scripts:
<property name="include.path" value=".\InstallScripts\targets" />
<property name="nant.contrib" value="c:\nant\contrib\bin\nant.contrib.tasks.dll" />

These properties are used for the build.include
<property name="solution.config" value="debug" />
<property name="compiler.target2003" value="C:\Program Files (x86)\Microsoft Visual Studio .NET 2003\Common7\IDE\devenv.exe" />
<property name="compiler.target2005" value="C:\Program Files (x86)\Microsoft Visual Studio 8\Common7\IDE\devenv.exe" />

These properties are set for ftp.include which also needs the ftp_mostrecent.bat file as well as the other parameters defined in the actual .include file for logging in.
<property name="ftp.executable" value="${include.path}\ftp_mostrecent.bat" />

These properties are set for prenis.include
<property name="prenis.executable" value="${include.path}\prenis.exe" />

These properties are set for svn.include and also you will require a command line version of subversion and also the GetAssemblyVersion.exe file which is included in the utilities pack.
<property name="svn.executable" value="C:\Program Files (x86)\CollabNet Subversion\svn.exe" />
<property name="svn.assemblyparser" value="${include.path}\GetAssemblyVersion.exe" />
<property name="svn.versionsrc" value="${include.path}" />
<property name="svn.username" value="username" />
<property name="svn.password" value="password" />
<property name="svn.baseuri" value="svn://subversion.pixolut.com/myrepository" />

Use the following to include the include files in your build script.


<include buildfile="${include.path}\ftp.include" />
<include buildfile="${include.path}\svn.include" />
<include buildfile="${include.path}\prenis.include" />
<include buildfile="${include.path}\build.include" />

Filed under: Open Source, pixolut, PreNIS, Software Development

PreNIS 1.1 Released

This version of the Nullsoft Install System Pre-Processor is built for .Net framework 2.0 and all source has been migrated to build under .Net 2.0 and Visual Studio 2005

UPDATES TO THIS VERSION FROM 1.0.3

+ 1.0.3 added the ability to have recursive includes – so that all
dependant DLLs could be grabbed from all projects which are used by
the referenced one. This concept has been extended to the macro
language so that you can dynamically choose to recurse projects.

+ We have added some new arguments for the macro.bin Macros – now
you can use the following:

includeProjects = [true|false]
* this will include all DLLs which are from the project(s).

includeReferences = [true|false]
* this will include referenced DLLs

recursive = [true|false]
* this will set the recursion through all dependancies capability.

Example:

##macro.bin;
assembly = Mobux.Web.Public;
includeProjects = true;
includeReferences = false;
##
File /nonfatal "%%absfilename%%"
##
macro.end
##

+ The final thing that we did in this release was make the script language
MUCH more tolerant of white space. As you can see in the above example you
can now split macros, imports or any command across lines and have as much
white space as you desire in your scripts. You can also use any case you
like with commands.

For more information on PreNIS you should go to the following places:

1. Learn about it at http://www.pixolut.com/prenis

2. Find out about NSIS http://nsis.sourceforge.net/

3. Contribute to the project at http://www.sourceforge.net/projects/prenis/

Filed under: Open Source, pixolut, PreNIS, Software Development

Getting Agile with BaseCamp

[37 Signals recently updated some great features on BaseCamp which fundamentally affect how we do Agile with our team, so I thought I would update this article as it still seems to get a lot of traffic.] 

After a lot of agony, I have found that there are really two ways of approaching a small lightweight agile development team with management tools. Rigid structure tools which try to implement a flavour of an agile methodology or loose structure tools which allow for your own approach to agility. There are actually plusses and minuses against each approach; so this statement is not about taking the steam out of Version One or Tackle by any means, but it is about our journey of discovery in dealing with non-technical customers and LOTS of disparate development tasks using only a thin veneer of project management on top.

We spent a lot of time looking at PMM tools like ProjectNet (which has a SCRUM module) and also dedicated agile management tools. We used Version One  for some time and whilst it was a JOY to use it really was focussed on a small area of scrum whilst overall project management was still left hanging. We just didn’t have enough management resource to deal with this upkeep of a micromanagement approach as well as all the project based high level demands of customers. We were also growing a lot (people and client wise) and V1 charge by user seat and it aint cheap for a company like mine – so with some regret we moved…

I tried in earnest to use Tackle, an open source alternative which is nice, but pretty immature. It shows in its lack of usability and it is still trying to enforce these approaches to Agile which we felt were hard to deal with.

After a brief attempt at using ProjectNet – which was inordinately complex to configure – we settled on BaseCamp. Its fees are not based on ‘per seat’ so the bigger you get the more cost effective it becomes. Its also very very flexible. Its not trying to impose a methodology on you, instead its got a very freeform approach to the architecture of a PMM tool and allows you to create the framework and the procedural rules to an extent.

Below is an introduction to how we use BaseCamp as a parrallel management tool for dealing with high level project mangement needs whilst also giving our developers the toosl they need to manage burn downs and stay agile…

1. Using MESSAGES for briefs (stories), bugs and support requests:
A brief for functionality, a bug request or a support ticket should be a MESSAGE entered in to BaseCamp with the appropriate category assigned to it. Once the brief is processed or completed we can change its state to completed by changing the category under which the item is filed. Just make sure that Admins set the MESSAGE status when you complete work. (ie: you need to have project managers or account directors or scrum masters with ADMIN access)

2. Using TODO Lists:
Each MILESTONE can have a TODO LIST. This TODO LIST should be set up to allow you to track time against it. By having a single goal with trackable time you can then start making the actual TODO ITEMS. Think of the TODO ITEMS for a goal as being like BurnDown chart for a scrum-oriented team goal. This allows an individual or mini team to manage their own development specific goals without making the high level project plan dirty.

This approach to using TODO LISTS also means that you can track your TODO LIST progress against a single Milestone which means no ‘Punting’ or moving all your tasks from week-to-week. Milestones can be set up like a sprint yet they can move if other demands come in without having to micromanage a shopping list of items you thought you could get done this week and now can’t.

3. Using MILESTONES:
MILESTONES give us the ability to create project goals and timings for our clients. Generally an Admin (Project Manager / Account Manager / Scrum Master) will create the MILESTONE after conference with developers regarding time and availability subject to customer priorities.

MILESTONES are the glue which bind together User Stories (MESSAGES) and Burn-Downs (TODO LISTS); once you create a MILESTONE for a User Story you can now bind the written brief (MESSAGE) to it. The TODO LIST allows the same function of selecting a MILESTONE and therefore viewing a MILESTONE will give the user visibility in to both the customer facing aspect and the developer facing aspect.

Depending on the technical ability of the customer, they can browse the TODO burndown or stay at the high level of MILESTONES and MESSAGES. They are connected yet (as mentioned above) they do not interfere with a simple high level understanding of a project.

Filed under: agile development, Software Development

New releases coming next week: PreNIS2 and BizBlox Builder

We have been busily migrating our core technologies from .Net 1.1 to .Net 2.0 – that is nearly two million lines of code across several projects and I have to say that it is testament to the Visual Studio 2005 environment that it has taken two developers less than a week to complete most of the migration. Increadible…

 Whilst we were at it, we found that there were some tools which had been left behind and needed a refresh:

 PreNIS, the NullSoft Install System pre-processor is going to version TWO next week. It now supports Visual Studio 2005 projects and also supports more advanced developer UI suites like Infragistics where they do not always ‘copy local’ all of the DLLs, but they are needed for web deployment.

The other product which desperately needed a refresh was the BizBlox Builder. Since the open source API went to version 1.4 it has supported Visual Studio 2005, however the Builder has not supported the newer SQL Schema of SQL2005 until now. You can now dynamically create thousands of lines of ORM code from the Builder using either SQL2000 or SQL2005 generated scripts.

Final testing is next week and releases will be up by then. If you’re really keen you can check out the current trunk on PreNIS to see what the new verison is capable of.

Filed under: BizBlox, Open Source, PreNIS, Software Development

Follow

Get every new post delivered to your Inbox.