Wednesday, March 4, 2015

Stonefield Query Enterprise Web Version 2.0 Released

Stonefield Software is pleased to announce the release of version 2.0 of Stonefield Query Enterprise Web. There are a dozens of new features in this release, the major ones being:

  • Charts and dashboards are now supported.
  • A new type of filtering is available: group filters. A regular filter condition filters at the record level; for example, show all customers with an invoice amount is greater than $100. A group filter condition filters at a group level; for example, show all customers whose total invoice amounts are greater than $100.
  • Embedded subreports that aren't linked to a field are now supported: click the new Subreport Properties button in step 5. This can be used, for example, to add a chart to a quick report.

See the online help for both Stonefield Query and Stonefield Query Studio for a complete list of new features.

For developers, note that version 2.0 has a breaking change if you’ve created your own plugins, as the interface for some plugins has changed. You’ll need to update and recompile your plugin code to work with version 2.0.

Thursday, February 12, 2015

Improving the Performance of Calculated Fields and Formulas, Part 2

In the previous blog post, I discussed one potential cause of slow performance of calculated fields and formulas: using the RunSQL function. In this post, I’ll discuss another common cause for performance issues: calling a script that retrieves multiple result sets from the database engine.

Note: this is an advanced topic intended for developers or technically-oriented users.

Calling a script from a formula

Sometimes, a formula needs to do more than just use an expression to calculate a value. In that case, it needs to call a script to execute some code. Often, that code retrieves additional data from the database, and that can lead to the same problem discussed in the previous blog post: multiple SQL statements sent to the database engine, slowing down performance.

For example, in Stonefield Query for Sage 300 (Sage 300 is an accounting system formerly known as Accpac), to determine the total quantity on hand for an inventory item, a script has to access the ICILOC table, which contains the quantity on hand at each location (such as each warehouse), and add up the various amounts for a particular item. Here’s a script that would do that:

lparameters tcItem
lcSelect = 'select sum(QTYONHAND) as QOH from ICILOC ' + ;
    'where ITEMNO=?tcItem'
loDB = SQApplication.DataEngine.Databases.GetMainDatabase()
loDB.ExecuteSQLStatement(lcSelect)
lnAmount = QOH
return lnAmount

If the name of this script is GetQOH, it would be used in a formula as follows:

GetQOH(ICITEM.ITEMNO)

The problem with this script is that if the report shows 1,000 inventory items, this code is called 1,000 times, and 1,000 additional SQL statements are sent to the database engine, which would likely be pretty slow.

The solution

The solution is to execute only a single SQL statement that sums the quantity on hand across all location for all items, not just the current one, keep that result set in memory (sometimes called caching), and then do an indexed lookup (a SEEK) to find the one we’re interested in. Here’s the code for the updated GetQOH script:

lparameters tcItem
if not used('_TotalQOH')
    lcSelect = 'select ITEMNO, sum(QTYONHAND) as QOH ' + ;
        '
from ICILOC group by ITEMNO'
    loDB = SQApplication.DataEngine.Databases.GetMainDatabase()
    loDB.ExecuteSQLStatement(lcSelect, , '_TotalQOH')
    index on ITEMNO tag KEY
endif
select _TotalQOH
seek tcItem
lnAmount = QOH
return lnAmount

There isn’t much extra code, but notice the SQL statement is only executed once, the first time the script is called, and that the result set it retrieves calculates the total for all items and indexes the result set on the item number. The second and subsequent times the script is called, it sees that the result set it needs is already in memory so it doesn’t bother retrieving it again, instead doing a SEEK to find the record for the current item. This can result in orders of magnitude improvement in performance, depending on the size of the result set retrieved for the report.

There are a couple of things to watch out for when writing a script that retrieves additional data.

Wrinkle #1: showing current data

After the report is run, the result set retrieved by the script is still cached. So, when you run the report again, it doesn’t bother retrieving a fresh set of data. That’s fine if the data doesn’t change but if users have added or edited records in the meantime and you want the most current data to be used, you have to tell Stonefield Query to close the cached data at the end of each report run. To do that, add the following line of code before the ENDIF statement in the code above:

oApp.oData.oCursorsToCloseAfterRun.Add('_TotalQOH')

Note: in the upcoming Stonefield Query version 5.1, this is a little simpler; you can use:

SQApplication.RegisterCursorToCloseAfterRun('_TotalQOH')

Wrinkle #2: handling multi-database reports

The second issue is a little more complicated, but only occurs if you have a multi-database report; that is, a report in which you’ve selected more than one database using the Database button on the Filter page (not all versions of Stonefield Query support this). Because Stonefield Query does all of the data retrieval it needs for the report before processing calculated fields and formulas, by the time your script is called, only the current database (the one shown in the status bar) is being used so the data your script retrieves comes from that database, not all of the databases for the report. In that case, your script has to handle getting the data from the correct database for the record it’s currently processing. A couple of things are required.

First, in order to know which database the record came from, you have to add the Source Database field to the report (this is a virtual field added to every table when multi-database queries are supported), even if you turn off its Display Field in Report setting so it doesn’t actually appear in the report. That way, your script can identify which database it needs to access to get the additional data for each record.

Second, your script has to retrieve more than one result set; it needs a different one from each database. That means it has to keep track of the current database, connect to the one used by the record if necessary, retrieve the data, and reconnect to the original database. Of course, we’ll cache the different result sets so this only happens once. Here’s the script above updated to support that:

lparameters tcItem
loDB = SQApplication.DataEngine.Databases.GetMainDatabase()
lcCurrent = loDB.CurrentDataSource.DataSource
if type('loReport') = 'O' and ;
    not empty(loReport.aDataSources[1]) and ;
    type('SOURCEDATABASE') = 'C'
    lcDataSource = trim(SOURCEDATABASE)
else
    lcDataSource = lcCurrent
endif
lcCursorSuffix = oUtility.GetValidName(lcDataSource)
lcCursor = '_TotalQOH' + lcCursorSuffix
if not used(lcCursor)
    llReopen = .F.

    if lcDataSource <> lcCurrent
        llReopen = .T.
        loDatabase.OpenDataSource(lcDataSource)
    endif

    lcSelect = 'select ITEMNO, sum(QTYONHAND) as QOH ' + ;
        'from ICILOC group by ITEMNO'
    loDB.ExecuteSQLStatement(lcSelect, , lcCursor)
    index on ITEMNO tag KEY
    if llReopen
        loDatabase.OpenDataSource(lcCurrent)
    endif
endif
select (lcCursor)
seek tcItem
lnAmount = QOH
return lnAmount

Whew! That’s a bit of complicated code, but you can use this example as a template for your own scripts. Just replace the values assigned to the lcCursor, lcSelect, and lnAmount variables, change the name of the variable used for the parameter and the SEEK statement (if desired; no harm if you leave it as “lcItem”), and change the INDEX ON statement to use the appropriate index expression.

Summary

Using the techniques in this blog post and the previous one can make your reports run significantly faster. They’re a bit of work to implement but the results are well worth the effort!

Improving the Performance of Calculated Fields and Formulas, Part 1

Stonefield Query has very fast database and calculation engines. Coupled with a high-performance database server like Microsoft SQL Server or Oracle, reports are often generated in Stonefield Query significantly faster than in other reporting solutions. We’ve heard of cases where a report took hours to run in (unnamed competitor) but only seconds in Stonefield Query.

However, occasionally we come across reports that run slower than expected. Frequently, the cause is a calculated field (defined by the developer in the data dictionary) or formula (defined by the user in the Formula Editor) that doesn’t perform well. This blog post and the next one discuss a couple of common causes and solutions. This post discusses the RunSQL function.

Note: this is an advanced topic intended for developers or technically-oriented users.

Using RunSQL

The RunSQL function has been in Stonefield Query for a long time. It was added to allow a formula to execute a SQL statement that retrieves additional data from the database. For example, suppose you want to show the amount of freight charged for each customer, but freight is a line item in an invoice details table rather than being a discrete field so you can’t just add a field for it to the report. In that case, a formula with an expression like this would calculate the freight charge for each customer:

RunSQL("select sum(AMOUNT) from INVOICE_DETAILS where ITEMNO = 'FREIGHT' and CUSTOMER_NO = ?CUST_ID")

The ?CUST_ID syntax means use the value of the CUST_ID field in the current record of the result set. In this case, the report displays the CUST_ID field from the CUSTOMERS table, then for each record, calculates the freight by executing the SQL statement in the RunSQL function.

The problem with RunSQL

The way Stonefield Query generates the result set for a report is as follows:

  • Determine the SQL statement needed to retrieve the data for “real” fields and for calculated fields and formulas that have the “Send formula to database engine” setting turned on.
  • Add to the SQL statement the fields necessary to perform the calculation of calculated fields and formulas that have the “Send formula to database engine” setting turned off.
  • Retrieve the data using the SQL statement into a temporary result set.
  • Go through every record in the temporary result set and process any calculated fields and formulas that have the “Send formula to database engine” setting turned off to give the final result set.

It’s the “go through every record” in the final point that causes the problem when you use RunSQL. Suppose you have 2,000 customers. As Stonefield Query processes the expression for the Freight formula, it sends a SQL statement to the database engine for every record. In this case, that’s 2,000 additional SQL statements. No wonder the report takes so long to run!

The solution

There are a couple of solutions to this problem. One is to use a new feature added in Stonefield Query version 5.0: grouping formulas. A grouping formula is like a regular formula but allows you to specify a summarization to be done on the expression (like the SUM function used in the SQL statement above), a field to group the summarized data on (like the implied grouping due to the filter on customer number in the SQL statement above), and optionally a filter to limit the records being retrieved (like the WHERE clause on ITEMNO in the SQL statement above). In many cases, a grouping formula can replace a formula using a RunSQL statement, but grouping formulas have one critical advantage: only a single SQL statement is sent to the database and the result set retrieved by it is cached and indexed for high-speed lookups. This results in orders of magnitude better performance, at the cost of using a little more memory.

GroupingFormula

Following the example above, you’d replace the expression of the Freight formula with just AMOUNT, click the Grouping button, select “Sum” for Summary, select CUST_ID for Grouping Field, and specify a filter of ITEMNO equals FREIGHT. The resulting formula is not only easier to create (you don’t have to know the SQL language or real table and field names) but performs a LOT faster.

For those few cases where a grouping formula can’t replace a RunSQL statement (although we haven’t encountered one yet), the other solution is to use the technique discussed in the next blog post: caching and indexing your own result set.

Thursday, November 27, 2014

Stonefield Query for HEAT Version 5.0

Stonefield Software Inc. is pleased to announce the release of version 5.0 of Stonefield Query for HEAT. This release has numerous new features.

  • Stonefield Query has a new report type: gauge report. This allows you to create a gauge showing the current value of something compared to a target or goal value.
  • You can now create a "grouping" formula. A grouping formula is a powerful tool that allows you to create filtered summary values. For example, suppose you want columns in a report that show the sales for last year and the current year-to-date for each customer. Those aren't single values retrieved from the database. Instead, for the first column, use a grouping formula that sums up the sales records for each customer for last year, and use a similar one but for the current year for the second column.
  • The Inventory Control Location Item Details table now also appears in the Purchase Orders module.
  • The Inventory Items table now has a Total Quantity on Hand field that contains the total quantity on hand from all locations.
  • The Inventory Shipments related tables no longer appear in the Order Entry or Purchase Orders modules to avoid confusion with the shipments tables already in those modules.
  • You can now password protect the file created when a report is output to file or email.
  • You can now print to Dymo LabelWriter labels.
  • The Schedule Report function is now available only to advanced users.
  • A new output type is now available: Microsoft Excel - Full Format (*.xlsx), which creates Excel 2007 or later files. The former Microsoft Excel (*.xls) output type was renamed to Microsoft Excel 97-2003 - Full Format (*.xls) so you can tell them apart. The new type does not require Microsoft Excel to be installed to create the file.
  • The "red" error dialog that appears when something goes wrong now has a Help button that brings up the help with information about that dialog.
  • The Locations page in the Options dialog is now available only to administrative users.
  • The Subscription Expired dialog, which appears when you have subscription licenses and your subscription has expired, now has a "Don't remind me again" option; turn that option on to prevent the dialog from appearing again. Also, those licenses are now deactivated rather than being deleted, so if you renew your subscription, they are reactivated.

Stonefield Query for ALERE Version 5.0

Stonefield Software Inc. is pleased to announce the release of version 5.0 of Stonefield Query for ALERE. This release has numerous new features.

  • Stonefield Query has a new report type: gauge report. This allows you to create a gauge showing the current value of something compared to a target or goal value.
  • You can now create a "grouping" formula. A grouping formula is a powerful tool that allows you to create filtered summary values. For example, suppose you want columns in a report that show the sales for last year and the current year-to-date for each customer. Those aren't single values retrieved from the database. Instead, for the first column, use a grouping formula that sums up the sales records for each customer for last year, and use a similar one but for the current year for the second column.
  • The Inventory Control Location Item Details table now also appears in the Purchase Orders module.
  • The Inventory Items table now has a Total Quantity on Hand field that contains the total quantity on hand from all locations.
  • The Inventory Shipments related tables no longer appear in the Order Entry or Purchase Orders modules to avoid confusion with the shipments tables already in those modules.
  • You can now password protect the file created when a report is output to file or email.
  • You can now print to Dymo LabelWriter labels.
  • The Schedule Report function is now available only to advanced users.
  • A new output type is now available: Microsoft Excel - Full Format (*.xlsx), which creates Excel 2007 or later files. The former Microsoft Excel (*.xls) output type was renamed to Microsoft Excel 97-2003 - Full Format (*.xls) so you can tell them apart. The new type does not require Microsoft Excel to be installed to create the file.
  • The "red" error dialog that appears when something goes wrong now has a Help button that brings up the help with information about that dialog.
  • The Locations page in the Options dialog is now available only to administrative users.
  • The Subscription Expired dialog, which appears when you have subscription licenses and your subscription has expired, now has a "Don't remind me again" option; turn that option on to prevent the dialog from appearing again. Also, those licenses are now deactivated rather than being deleted, so if you renew your subscription, they are reactivated.

Stonefield Query for AccountMate Version 5.0

Stonefield Software Inc. is pleased to announce the release of version 5.0 of Stonefield Query for AccountMate. This release has numerous new features.
  • AccountMate Version 9.2 is now supported.
  • Stonefield Query has a new report type: gauge report. This allows you to create a gauge showing the current value of something compared to a target or goal value.
  • You can now create a "grouping" formula. A grouping formula is a powerful tool that allows you to create filtered summary values. For example, suppose you want columns in a report that show the sales for last year and the current year-to-date for each customer. Those aren't single values retrieved from the database. Instead, for the first column, use a grouping formula that sums up the sales records for each customer for last year, and use a similar one but for the current year for the second column.
  • The Inventory Control Location Item Details table now also appears in the Purchase Orders module.
  • The Inventory Items table now has a Total Quantity on Hand field that contains the total quantity on hand from all locations.
  • The Inventory Shipments related tables no longer appear in the Order Entry or Purchase Orders modules to avoid confusion with the shipments tables already in those modules.
  • You can now password protect the file created when a report is output to file or email.
  • You can now print to Dymo LabelWriter labels.
  • The Schedule Report function is now available only to advanced users.
  • A new output type is now available: Microsoft Excel - Full Format (*.xlsx), which creates Excel 2007 or later files. The former Microsoft Excel (*.xls) output type was renamed to Microsoft Excel 97-2003 - Full Format (*.xls) so you can tell them apart. The new type does not require Microsoft Excel to be installed to create the file.
  • The "red" error dialog that appears when something goes wrong now has a Help button that brings up the help with information about that dialog.
  • The Locations page in the Options dialog is now available only to administrative users.
  • The Subscription Expired dialog, which appears when you have subscription licenses and your subscription has expired, now has a "Don't remind me again" option; turn that option on to prevent the dialog from appearing again. Also, those licenses are now deactivated rather than being deleted, so if you renew your subscription, they are reactivated.

Monday, November 17, 2014

Stonefield Query for Sage Pro ERP Version 5.0

Stonefield Software Inc. is pleased to announce the release of version 5.0 of Stonefield Query for Sage Pro ERP. This release has numerous new features.

  • Stonefield Query has a new report type: gauge report. This allows you to create a gauge showing the current value of something compared to a target or goal value.
  • You can now create a "grouping" formula. A grouping formula is a powerful tool that allows you to create filtered summary values. For example, suppose you want columns in a report that show the sales for last year and the current year-to-date for each customer. Those aren't single values retrieved from the database. Instead, for the first column, use a grouping formula that sums up the sales records for each customer for last year, and use a similar one but for the current year for the second column.
  • You can now password protect the file created when a report is output to file or email.
  • You can now print to Dymo LabelWriter labels.
  • The Schedule Report function is now available only to advanced users.
  • A new output type is now available: Microsoft Excel - Full Format (*.xlsx), which creates Excel 2007 or later files. The former Microsoft Excel (*.xls) output type was renamed to Microsoft Excel 97-2003 - Full Format (*.xls) so you can tell them apart. The new type does not require Microsoft Excel to be installed to create the file.
  • The "red" error dialog that appears when something goes wrong now has a Help button that brings up the help with information about that dialog.
  • The Locations page in the Options dialog is now available only to administrative users.
  • The Subscription Expired dialog, which appears when you have subscription licenses and your subscription has expired, now has a "Don't remind me again" option; turn that option on to prevent the dialog from appearing again. Also, those licenses are now deactivated rather than being deleted, so if you renew your subscription, they are reactivated.

Stonefield Query for Sage 300 ERP Version 5.0

Stonefield Software Inc. is pleased to announce the release of version 5.0 of Stonefield Query for Sage 300 ERP. This release has numerous new features.

  • Stonefield Query has a new report type: gauge report. This allows you to create a gauge showing the current value of something compared to a target or goal value.
  • You can now create a "grouping" formula. A grouping formula is a powerful tool that allows you to create filtered summary values. For example, suppose you want columns in a report that show the sales for last year and the current year-to-date for each customer. Those aren't single values retrieved from the database. Instead, for the first column, use a grouping formula that sums up the sales records for each customer for last year, and use a similar one but for the current year for the second column.
  • The Inventory Control Location Item Details table now also appears in the Purchase Orders module.
  • The Inventory Items table now has a Total Quantity on Hand field that contains the total quantity on hand from all locations.
  • The Inventory Shipments related tables no longer appear in the Order Entry or Purchase Orders modules to avoid confusion with the shipments tables already in those modules.
  • You can now password protect the file created when a report is output to file or email.
  • You can now print to Dymo LabelWriter labels.
  • The Schedule Report function is now available only to advanced users.
  • A new output type is now available: Microsoft Excel - Full Format (*.xlsx), which creates Excel 2007 or later files. The former Microsoft Excel (*.xls) output type was renamed to Microsoft Excel 97-2003 - Full Format (*.xls) so you can tell them apart. The new type does not require Microsoft Excel to be installed to create the file.
  • The "red" error dialog that appears when something goes wrong now has a Help button that brings up the help with information about that dialog.
  • The Locations page in the Options dialog is now available only to administrative users.
  • The Subscription Expired dialog, which appears when you have subscription licenses and your subscription has expired, now has a "Don't remind me again" option; turn that option on to prevent the dialog from appearing again. Also, those licenses are now deactivated rather than being deleted, so if you renew your subscription, they are reactivated.

Thursday, November 13, 2014

Stonefield Query for Act! Version 5.0

Stonefield Software Inc. is pleased to announce the release of version 5.0 of Stonefield Query for Act!. This release has numerous new features.

  • Stonefield Query is compatible with the new Act! v17 (2015) and Act! 2014 Service Pack 1 (16.1).
  • Stonefield Query has a new report type: gauge report. This allows you to create a gauge showing the current value of something compared to a target or goal value.
  • You can now create a "grouping" formula. A grouping formula is a powerful tool that allows you to create filtered summary values. For example, suppose you want columns in a report that show the sales for last year and the current year-to-date for each customer. Those aren't single values retrieved from the database. Instead, for the first column, use a grouping formula that sums up the sales records for each customer for last year, and use a similar one but for the current year for the second column.
  • You can now password protect the file created when a report is output to file or email.
  • You can now print to Dymo LabelWriter labels.
  • The Schedule Report function is now available only to advanced users.
  • A new output type is now available: Microsoft Excel - Full Format (*.xlsx), which creates Excel 2007 or later files. The former Microsoft Excel (*.xls) output type was renamed to Microsoft Excel 97-2003 - Full Format (*.xls) so you can tell them apart. The new type does not require Microsoft Excel to be installed to create the file.
  • The "red" error dialog that appears when something goes wrong now has a Help button that brings up the help with information about that dialog.
  • The Locations page in the Options dialog is now available only to administrative users.
  • The Subscription Expired dialog, which appears when you have subscription licenses and your subscription has expired, now has a "Don't remind me again" option; turn that option on to prevent the dialog from appearing again. Also, those licenses are now deactivated rather than being deleted, so if you renew your subscription, they are reactivated.

Stonefield Query for GoldMine Version 5.0

Stonefield Software Inc. is pleased to announce the release of version 5.0 of Stonefield Query for GoldMine. This release has numerous new features.

  • Stonefield Query has a new report type: gauge report. This allows you to create a gauge showing the current value of something compared to a target or goal value.
  • You can now create a "grouping" formula. A grouping formula is a powerful tool that allows you to create filtered summary values. For example, suppose you want columns in a report that show the sales for last year and the current year-to-date for each customer. Those aren't single values retrieved from the database. Instead, for the first column, use a grouping formula that sums up the sales records for each customer for last year, and use a similar one but for the current year for the second column.
  • You can now password protect the file created when a report is output to file or email.
  • You can now print to Dymo LabelWriter labels.
  • The Schedule Report function is now available only to advanced users.
  • A new output type is now available: Microsoft Excel - Full Format (*.xlsx), which creates Excel 2007 or later files. The former Microsoft Excel (*.xls) output type was renamed to Microsoft Excel 97-2003 - Full Format (*.xls) so you can tell them apart. The new type does not require Microsoft Excel to be installed to create the file.
  • The "red" error dialog that appears when something goes wrong now has a Help button that brings up the help with information about that dialog.
  • The Locations page in the Options dialog is now available only to administrative users.
  • The Subscription Expired dialog, which appears when you have subscription licenses and your subscription has expired, now has a "Don't remind me again" option; turn that option on to prevent the dialog from appearing again. Also, those licenses are now deactivated rather than being deleted, so if you renew your subscription, they are reactivated.