News, examples, tips, ideas and plans.
Thoughts around ORM, .NET and SQL databases.

Friday, December 27, 2013

Winter sale

This year we're starting our winter sale right on December 27. Just use the coupon code DO2014 on checkout and get 50% off all editions of DataObjects.Net. If you want to renew your subscription use DO2014RENEWAL to get 70% discount.

This special on our products is valid until Wednesday 15 January 2014.

Should you have any questions, please email to sales@x-tensive.com.

Sunday, November 17, 2013

DataObjects.Net 5.0 Beta 1

This is a short announce about the upcoming major version of DataObjects.Net.

Changes:

Added

  • PostSharp is replaced with custom persistence weaver based on Mono.Cecil
  • Support for String.Like extension method with support in LINQ translator
  • Support for DateTime.ToString("s") in LINQ
  • Support for ignoring particular tables and/or columns during Domain.Build
  • Support for defining recycled fields via external definitions
  • ReSharper annotations for public API

Changed

  • Xtensive.Tuples performance is significantly improved
  • Validation framework is reworked
  • Persist operation is optimized to avoid sorting entities that do not have incoming/outgoing FK
  • IEnumerable.Remove() extension method is marked as obsolete

Removed

  • Support for .NET 3.5 and Visual Studio 2008
  • Object-to-object mapper
  • Xtensive.Core and Xtensive.Aspects assemblies. Now we have single Xtensive.Orm assembly with a bunch of database drivers in separate assemblies.
  • Installer. It is no longer provided so use NuGet or binaries package instead

Download

Just as always, releases are available from NuGet gallery.

Avoiding connection pool fragmentation


Microsofties admitted that some noticeable performance problems are possible while working with MS SQL Server via ADO.Net (link, scroll to Pool Fragmentation). The connection pooling technique used in ADO.Net to optimize and minimize the cost of opening connections reported to be "not so optimal" in particular scenarios.

The root of the problem is that the exact match between connection string and corresponding connection pool is required. Even if 2 connection strings are build from the same key-value pairs but differ in their order, you get 2 connection pools.

As a result, we get connection pool fragmentation, which is a common problem in many Web applications where the application can create a large number of pools that are not freed until the process exits. This leaves a large number of connections open and consuming memory, which results in poor performance. Surprise for large multi-tenant application developers and maintainers!

From their side, Microsoft is not going to do anything about it in foreseeable future, instead they suggest to use a workaround: connect to master database and open the desired database with a separate command:

// Assumes that command is a SqlCommand object and that
// connectionString connects to master.
command.Text = "USE DatabaseName";
using (SqlConnection connection = new SqlConnection(
  connectionString))
  {
    connection.Open();
    command.ExecuteNonQuery();
  }

How can you achieve the same not leaving the zone of comfortable DataObjects.Net API? Time to get familiar with new member of DomainConfiguration class: ConnectionInitializationSql. The main purpose of the member is to literally execute the provided text as DbCommand right after a connection is opened.

Say, we are using Northwind database:


<domain
name="Default"
provider="sqlserver"
connectionString="Server=myServerAddress;Database=master;..."
connectionInitializationSql="USE NORTHWIND"
...>
<domain>


The feature is available in DataObjects.Net 4.6.4. Download DataObjects.Net

Thursday, November 14, 2013

Support for in-memory database

I'm not sure whether anyone remembers, but in the beginning of DataObjects.Net 4.0 epoch we had a semi-successful attempt to provide our own in-memory database implementation, which existed for several years but eventually was removed from the product by various reasons.

And guess what? Starting from DataObjects.Net 4.6.4, in-memory database support is back! But this time it is not our own version, but SQLite in-memory mode.

To connect to in-memory database use the following connection string:

<domain
  name="Default"
  provider="sqlite"
  connectionString="Data Source=:memory:"
  ...>
<domain>

The advantages of such kind of storage are the highest possible speed of database-related operations and no necessity to bother with database files on disk. On the contrary, there are also disadvantages, like the absence of multiple connections to database and no real persistence.

SQLite in-memory mode has some requirements that DataObjects.Net must have met. The most important one is that as soon as connection to in-memory database is closed, the database is destroyed and all data is lost.

Before version 4.6.4, DataObjects.Net always built database scheme in a separate connection which was closed after that procedure. Moreover, the ORM didn't keep any connections open at all, closing them as soon as corresponding session is disposed to free up resources and return connection to connection pool. To support the SQL in-memory mode we had to change the connection management layer and make it more flexible and configurable.

And so we did it. Now we open connection to SQL in-memory database only once on domain build and keep it open until domain is disposed, protecting the database from being destroyed, so your data is safe while domain is alive. In the meantime, you may open and close sessions as usual except you shouldn't open concurrent sessions as only one connection is allowed at a time.

Bad practice: using nested sessions with SQLite in-memory database. Session use the same connection concurrently.

  using (var session1 = domain.OpenSession()) {
    using (var t1 = session1.OpenTransaction()) {

      using (var session2 = domain.OpenSession()) {
        // Here you'll get InvalidOperationException 
        // that the connection is used by another session


Good practice: using subsequent sessions. Both sessions use the same connection, but not concurrently.

  using (var session1 = domain.OpenSession()) {
    using (var t1 = session1.OpenTransaction()) {

      // Do some stuff
      t1.Complete();
    }
  }

  using (var session2 = domain.OpenSession()) {
    using (var t2 = session2.OpenTransaction()) {

      // Do some stuff
      t2.Complete();
    }
  }

Tuesday, November 12, 2013

Enhancement in optimistic concurrency mode

Starting from DataObjects.Net 4.6.4 we are introducing an update to optimistic concurrency feature — server-side version check.

The present API of optimistic concurrency feature in DataObjects.Net with all these VersionSet, VersionCapturer, VersionValidator, whatever is kind of tricky, over-complicated and mind-blowing, so eventually we started moving towards more simple and transparent solution. The server-side version check is the first step in this direction.

The whole idea of server-side version check is obvious: each time an entity is fetched from database, it is associated with a version. On each successful update version is incremented. Each UPDATE command contains additional check for the specific version. Here is an example:

Say, we operate an online book store and use the following simplified Book model with Version field for optimistic concurrency check:

[HierarchyRoot]
public class Book : Entity
{
    [Field, Key]
    public int Id { get; set; }

    [Field(Length = 128)]
    public string Title { get; set; }

    [Field, Version]
    public int Version { get; set; }

    public Book(Session session) : base(session)
    {}
}

As the field is marked with VersionAttribute, DataObjects.Net detects and uses it to store the Book version which is automatically incremented after each successfully committed transaction. Now we want that every update command to include the additional check for version, e.g.:

UPDATE [dbo].[Book]  
SET [Title] = 'DataObjects.Net 4 unleashed' 
WHERE (([Book].[Id] = 123) AND ([Book].[Version] = 2)); 

If this check fails, the Xtensive.Orm.VersionConflictException is thrown with message "Version of entity with key 'Book, (123)' differs from the expected one", so it can be easily detected and handled.

By default, this mode in DataObjects.Net 4.6.4. is switched off to provide compatibility with the older versions. To switch it on we should change SessionConfiguration like this:

var sessionConfig = new SessionConfiguration(
    SessionOptions.ServerProfile | SessionOptions.ValidateEntityVersions);

using (var session = domain.OpenSession(sessionConfig)) {
    // do some stuff 
}

Alternatively, this can be done in configuration file so it is automatically applied to all sessions, like this:

<Xtensive.Orm>
    <domains>
      <domain name="Default"
              upgradeMode="Recreate"
              connectionUrl="sqlserver://localhost/AmazonBookStore">
        ...
        <sessions>
          <session name="Default" options="ServerProfile, ValidateEntityVersions" />
        </sessions>
      </domain>
    </domains>
  </Xtensive.Orm>

Monday, September 30, 2013

DataObjects.Net 4.5.8 and 4.6.4

Finally, the new version of DataObjects.Net is out. Traditionally, we are releasing both major versions: 4.5.x & 4.6.x,

Changes in both DataObjects.Net 4.5.8 and 4.6.4:


[main] Added QueryEndpoint.Items() method for querying EntitySets within compiled queries
[main] Added support for DateTime.AddXxx() methods in LINQ translator
[main] Added support for System.Linq.EnumerableQuery in LINQ translator
[main] Fixed invalid key comparison under certain conditions
[main] Fixed EntitySet caching invalid state within DisableSaveChanges() scope
[main] Fixed incorrect column references in generated SQL for certain queries
[main] Fixed querying for types with enum type discriminators
[main] Fixed querying for types with enum key fields
[main] Fixed locking of entity that could not be persisted at the moment for some reason
[main] Fixed handling of temporary tables query with zero columns
[main] Fixed translation of GroupJoin() with temporary tables
[main] Fixed translation of Distinct() after Select()
[main] Fixed translation of ThenBy() that implicitly adds joins
[main] Fixed support for LINQ member compilers that return nullable values
[main] Fixed translation of as with subquery operand
[main] Fixed concurrent access issues with NameBuilder.GetHash method
[main] Fixed incorrect result of FirstOrDefault/SingleOrDefault in certain subqueries
[main] Optimized translation of String.IsNullOrEmpty() method
[main] Reduced number of casts in generated SQL when accessing enum fields
[main] Automatically handle namespace-only renames during upgrade
[main] PostSharp is upgraded to version 2.1.7.30
[postgresql] Npgsql is upgraded to version 2.0.12.1
[mysql] MySQL library is upgraded to version 6.7.4
[mysql] Fixed translation of bitwise operations
[mysql] Fixed translation of DateTime.DayOfWeek and DateTime.DayOfYear
[firebird] Firebird library is upgraded to version 3.0.2.1
[sqlserver] Fixed reading of large SqlDecimal values

Changes specific to DataObjects.Net 4.6.4:

[main] Added version validation on persist via SessionOptions.ValidateEntityVersions
[main] Added DomainConfiguration.ConnectionInitializationSql option
[main] Added support for Enum.HasFlags method in LINQ
[sqlite] Added support for :memory: data source
[main] Fixed regression in Session.Query.ExecuteDelayed introduced in 4.6.4 RC
[main] Fixed redundant delete queries when clearing key generator tables during upgrade
[main] Fixed NRE in StorageMappingBuilder when persistent type does not have a namespace
[main] Store partial index filter definitions in Metadata.Extension table instead of relying on information schema
[main] Improved diagnostic messages for multimapping configurations when database/schema is not found
[main] Improved diagnostic message when non-LINQ method is called within Session.Query.ExecuteDelayed
[sqlite] Fixed extraction of multi-column primary keys
[sqlite] SQLite library is upgraded to version 1.0.87.0
[sqlserver] Fixed incorrect SQL for table rename in non-default database

Download

Just as always releases are available at our site and NuGet gallery.

Tuesday, February 05, 2013

DataObjects.Net in 2013 roadmap

This post will cover our plans for DataObjects.Net in year 2013.

Feature update for DataObjects.Net 4.6

Unlike typical bug-fix-only minor releases. DataObjects.Net 4.6.4 will include
several new features.

Support for ignoring certain tables and columns during upgrade

This is essential feature for using technologies such as SQL Server Replication that require special columns to be present in each table.

Support for external 'recycled' definitions

As you know, it's possible to mark certain types and/or fields as recycled to keep corresponding data (tables, columns) available during upgrade. However, this requires keeping some legacy items in your code. External recycled definition will solve this problem by providing an API to add recycled definitions via UpgradeHandlers.

Support for :memory: data source in SQLite provider

SQLite provides special :memory: data source that could be used to operate against in-memory database. This is extremely useful for instance, for testing purposes as the database disappears once connection is closed. Currently, DataObjects.Net requires separate connections for building domain and regular CRUD operations which makes :memory: data source inapplicable. DataObjects.Net 4.6.4 will include support for this data source, limiting one active session per domain.

DataObjects.Net 4.6.4 is expected to be released in March 2013.

Plans for DataObjects.Net 4.7

This is not final roadmap for DataObjects.Net 4.7. Things might change, but here is our current plans.

New faster tuple implementation

DataObjects.Net internally uses special data structure to store Entity fields called Tuple (not to be confused with .NET Framework 4 tuples). Versions prior to 4.7 used set of generic types to store and access tuple fields. Due to the startup slowness of generics and additional memory consumption we switched to non-generic approach that provides more efficient data storage as well as fast typed access. This feature is already implemented and will be included in DataObjects.Net 4.7

Support for reading entities after session has been disposed

This feature would simplify passing entities in ASP.NET MVC and other similar applications.

Automatic caching of generated SQL queries

LINQ translation might take considerable time. DataObjects.Net provides compiled queries to address this. However sometimes it's desirable to cache translation result of each LINQ query. This would be available as an option. By default generated SQL queries will not be cached.

Query result caching API

This is one of the most wanted DataObjects.Net features. We will provide an API to maintain cache of LINQ query results as well as examples how to integrate DataObjects.Net with memcached and other caching facilities.

Change of a default Transactional aspect application

At the moment all your persistent types (i.e. entities, services derived from SessionBound) are powered with our Transactional aspect. This means they automatically provide transaction and activate session upon calling any of their methods. This adds some overhead and most of the users like to manage transactions manually. Thus we decided to remove automatic application of  Transactional aspect in DataObjects.Net 4.7. For those who will need current behavior it would be possible to enable it again by adding special attribute on each assembly with persistent types.

Improved support for native SQL data types

Currently DataObjects.Net supports persistence of standard .NET primitive types and spatial types of SQL Server. We are going to improve this and add support for spatial types in PostgreSQL as well as XML data types for servers that support them. Also operations on such types would be supported by LINQ translator.

Improved logging API

At the moment DataObjects.Net logging is complex to configure.
We're going to change this and provide new more simple interface for it. Also it would be possible to plug your own log consumers.

Reworked validation framework

We're going to improve our validation framework to make it more easy to use and extend with your own validators.

Support for SQL Server Compact 4

SQL Server Compact 4.0 will supported in addition to 3.5

DataObjects 4.7 is scheduled for release in September 2013.
 

Monday, February 04, 2013

Product lifetime policy update

There was no clear policy on DataObjects.Net product lifetime in the past. Now we define such policy. Each major DataObjects.Net release is supported for 18 months. Within this period we will accept bug reports and make bug-fix releases. After product lifetime expires you might still use it "as is". Certain versions might have this period expanded with proper announcement.

Currently we support 3 major DataObjects.Net releases: 4.4, 4.5 and 4.6.

The following table shows when they would go out of support:
  • 4.4    April 2013
  • 4.5    October 2013
  • 4.6    April 2014
As a special exception we extended 4.4 lifetime to make it easy to migrate to newer versions.
The last 4.4 release will be available in April 2013.

DataObjects.Net 4.5.7 and 4.6.3 are released

This is a huge bug-fix release. Everybody is encouraged to upgrade.

DataObjects.Net 4.5.7 and 4.6.3:

DataObjects.Net 4.6.3 only:

  • Fixed incorrect column order in queries after certain schema upgrades
  • Added DomainConfiguration.NativeLibraryCacheFolder setting
  • Significantly improved initialization time for Xtensive.Tuples infrastructure

DataObjects.Net Extensions 4.6.3 and 4.5.7:

Bulk operations extension
  •  Remove extra command execution under certain scenarios
Reprocessing extension
  • Improve compatibility with DisconnectedState
  • Don't use nullable types in ReprocessAttribute properties

DataObjects.Net Extensions 4.6.3 only:

Localization extension
  • Improved examples in readme.txt
Tracking extension
  • Fixed incorrect handling of partially loaded entities

DataObjects.Net LinqPad driver 4.6:

Download

Just as always releases are available at our site and NuGet gallery.

Monday, November 19, 2012

DataObjects.Net 4.5 and 4.6 are updated

Bug fixing updates for DataObjects.Net 4.5 and 4.6 are available.

Here is the list of changes for both versions:
  • Fix translation of predicates similar to (object) NullableBool == (object) null
  • Fix translation of Union() over boolean columns under certain conditions
  • Don't enforce implicit non-nullable/length constraints in inconsistent sessions
  • Import DataObjects.Net.targets conditionally when using NuGet package
  • Make UpgradeHandler.IsFieldAvailable virtual
  • Treat SQL Server errors 3966 and 3971 as serialization failure

The following fixes apply to 4.6.1 only:
  • Fix loading/storing of the domain model when multi-database mode is enabled
  • Fix assignment of MappingSchema/MappingDatabase for persistent interfaces
  • Improve exception message for missing default schema in domain configuration
Just as always you can get new releases at our download site or NuGet gallery.

DataObjects.Net extensions are not updated yet. You could use previous versions of them which are fully compatible with this release.