DataObjects.Net 4.6.6

Changes in DataObjects.Net 4.6.6:

[main] Added DomainConfiguration.MultidatabaseKeys setting
[main] Fixed incorrect EntitySet caching in ClientProfile
[main] Fixed construction of inherited partial indexes
[main] Fixed NRE when using partial indexes over abstract types
[main] Fixed incorrect type id upgrade when type is moved to another namespace
[oracle] Exclude many standard schemes during extraction

Download

Just as always releases are available at our site and NuGet gallery.

NonTransactionalReads in DataObjects.Net 5.0

NonTransactionalReads is a specific Session option that allows to read once fetched entities right from Session cache without the requirement to have an open transaction. This mode might be useful for desktop application that have only one Session instance for the entire lifetime of application or for services like auditing, logging, etc. that need to read persistent fields without having any idea of sessions, transactions, etc.

To demonstrate the approach, let's start with a sample. Here is the persistent class with one lazy loading field:

[HierarchyRoot]
public class Country : Entity
{
    [Field, Key]
    public int Id { get; private set; }

    [Field(Length = 30)]
    public string Name { get; set; }

    [Field(Length = 3)]
    public string Code { get; set; }

    [Field(LazyLoad = true)]
    public string Description { get; set; }

    public Country(Session session)
        : base(session) {}
}

Say, we want to have a cache of countries in our application. We create some countries on Domain start, e.g.:

var domainConfiguration = DomainConfiguration.Load("Default");
var domain = Domain.Build(domainConfiguration);

using (var session = domain.OpenSession())
using (var tx = session.OpenTransaction()) {
    new Country(session) {
        Code = "AND",
        Name = "Andorra",
        Description = "A tiny country in the middle of Europe"
    };
    new Country(session) {
        Code = "ATA",
        Name = "Antarctica",
        Description = "Icy paradise for penguins"
    };
    // other countries...
    tx.Complete();
}

To implement some sort of cache in earlier version of DataObjects.Net you would load countries in every transaction or avoid frequent loading of persistent objects from database by converting them into some kind of POCO and cache them.

In DataObjects.Net 5.0 you can do this in a more comfortable way using the new NonTransactionalReads flag.

// Configuring session
var options = new SessionConfiguration(SessionOptions.Default | SessionOptions.NonTransactionalReads);

var session = domain.OpenSession(options);
Dictionary<string, Country> cache;

// Loading cache with data in one transaction
using (var tx = session.OpenTransaction()) {
    cache = session.Query.All<Country>().ToDictionary(i => i.Code);
    tx.Complete();
}

// accessing data from another transaction
using (var tx2 = session.OpenTransaction()) {

    // cached objects are not being reloaded from database. they are consumed as is
    var antarctica = cache["ATA"];
    Console.WriteLine(antarctica.Name);

    // accessing a lazy loading field. session loads it on demand
    Console.WriteLine(antarctica.Description);

    tx2.Complete();
}

// disposing session. Data is not accessible anymore
session.Dispose();

What is more interesting, the same behavior can be achieved without using any transactions at all. Let's re-write the sample:

// Configuring session
var options = new SessionConfiguration(SessionOptions.Default | SessionOptions.NonTransactionalReads);

var session = domain.OpenSession(options);
Dictionary<string, Country> cache;

// Loading cache with data without any transaction
cache = session.Query.All<Country>().ToDictionary(i => i.Code);

// accessing data 
var antarctica = cache["ATA"];
Console.WriteLine(antarctica.Name);

// accessing a lazy loading field. session loads it on demand
Console.WriteLine(antarctica.Description);

// disposing session. Data is not accessible anymore
session.Dispose();

So, the key points of the NonTransactionalReads mode:

1. No matter how data is loaded from database (with the help of transaction or not), it is accessible from outside the boundaries of the transaction, if any.
2. Data is cached on Session level. As long as the Session is alive (not disposed), the data will be available from its cache.
3. In case of accessing LazyLoad fields, Entity references and EntitySets, data is automatically fetched from database on demand.

DataObjects.Net 5.0 meets NLog & log4net

In the previous post we explained DataObjects.Net built-in logging capabilities. Now it is time to show how popular logging libraries like log4net and NLog can be integrated with it.

We would need the list of names of built-in logs:

Xtensive.Orm - logs Session & Transaction-related events and exceptions.

Xtensive.Orm.Building - logs events during the Domain building process.

Xtensive.Orm.Sql - logs SQL statements sent to database server.

Xtensive.Orm.Upgrade - logs events during database schema upgrade.

log4net

1. Add DataObjects.Net logging provider for log4net. It will automatically add log4net.
2. Set up log provider in Xtensive.Orm configuration section
   
   

    <Xtensive.Orm>
      <domains>
        <domain ... >
        </domain>
      </domains>
      <logging provider="Xtensive.Orm.Logging.log4net.LogProvider, Xtensive.Orm.Logging.log4net">
    </Xtensive.Orm>
   

3. Configure log4net. Use the above-mentioned loggers' names, e.g.:
   
   

    <?xml version="1.0" encoding="utf-8" ?>
    <configuration>
      <configSections>
        ...
        <section name="log4net" type="log4net.Config.Log4NetConfigurationSectionHandler,log4net"/>
      </configSections>
    
      <log4net>
        <appender name="ConsoleAppernder" type="log4net.Appender.ConsoleAppender">
          <layout type="log4net.Layout.PatternLayout">
            <conversionPattern value="%date [%thread] %-5level %logger %message%newline" />
          </layout>
        </appender>
      
        <logger name="Xtensive.Orm">
          <level value="ALL" />
          <appender-ref ref="ConsoleAppernder" />
        </logger>
      </log4net>
    </configuration>
   
   

NLog

1. Add DataObjects.Net logging provider for NLog. It will automatically add NLog.
2. Set up log provider in Xtensive.Orm configuration section
   
   

    <Xtensive.Orm>
      <domains>
        <domain ... >
        </domain>
      </domains>
      <logging provider="Xtensive.Orm.Logging.NLog.LogProvider, Xtensive.Orm.Logging.NLog">
    </Xtensive.Orm>
   

3. Configure NLog. Use the above-mentioned loggers' names, e.g.:
   
   

       <?xml version="1.0" encoding="utf-8" ?>
       <nlog xmlns="http://www.nlog-project.org/schemas/NLog.xsd"
             xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance">
   
         <targets>
           <target name="console" xsi:type="Console" />
         </targets>
   
         <rules>
           <logger name="Xtensive.Orm" minlevel="Debug" writeTo="console" />
         </rules>
       </nlog>
   

Pretty simple, huh? You can even write your own log provider with ease. Just take a look at the implementation of these two in DataObjects.Net Extensions repository, default branch.

DataObjects.Net 5.0 built-in logging

Yeah, we have to admit it - logging in DataObjects.Net 4.x was implemented quite odd and its API was not clear even to some of product developers, not to mention customers who were wondering whether the bloody logging works at all and how the hell it is supposed to be configured. Finally, these days are gone and starting from DataObjects.Net 5.0 we re-implemented the logging.

So, what has changed? The answer is - almost all, including inconsistent loggers' names, tricky xml configuration and peculiar log formats. Imho, the best characteristic for the new logging will be "brain-dead obvious" "extremely simple".

OK, let's dive into the details.

Loggers

These are named logs that record messages from specific parts of DataObjects.Net. There are 4 of them:

Xtensive.Orm - logs Session & Transaction-related events and exceptions.

Xtensive.Orm.Building - logs events during the Domain building process.

Xtensive.Orm.Sql - logs SQL statements sent to database server.

Xtensive.Orm.Upgrade - logs events during database schema upgrade.

Log writers

Console - writes messages to application's console window, if any. Useful for small & test projects.

DebugOnlyConsole - the same as Console but writes log data only when a project is run in Debug mode.

path_to_file - appends log messages to a specified file. If file is absent, it will be created. Useful for development & production environments. path_to_file can be either absolute or relative to the application location.

None - writes to /dev/null.

Configuration

Configuration of built-in logging is made in application configuration file (app.config or web.config). Logger configuration takes 2 parameters: logger name as source and log writer name as target.

Example: Logging everything to Console

  <Xtensive.Orm>
    <domains>
      <domain name="Default".../>
    </domains>

    <logging>
      <log source="Xtensive.Orm" target="Console"/>
      <log source="Xtensive.Orm.Building" target="Console"/>
      <log source="Xtensive.Orm.Sql" target="Console"/>
      <log source="Xtensive.Orm.Upgrade" target="Console"/>
    </logging>
  </Xtensive.Orm>

Example: Logging everything to files

  <Xtensive.Orm>
    <domains>
      <domain name="Default".../>
    </domains>

    <logging>
      <log source="Xtensive.Orm" target="C:\Orm.log"/>
      <log source="Xtensive.Orm.Building" target="C:\Orm.Building.log"/>
      <log source="Xtensive.Orm.Sql" target="C:\Orm.Sql.log"/>
      <log source="Xtensive.Orm.Upgrade" target="C:\Orm.Upgrade.log"/>
    </logging>
  </Xtensive.Orm>

To simplify things in case you want to write messages from all loggers into the same stream, you may want to use our magic asterisk, like this: Example: Logging everything to a file

  <Xtensive.Orm>
    <domains>
      <domain name="Default".../>
    </domains>

    <logging>
      <log source="*" target="C:\Orm.log"/>
    </logging>
  </Xtensive.Orm>

In addition to the built-in log writers DataObjects.Net 5 is shipped with 2 extensions allowing to redirect logging output to NLog & log4net. Read about that in the second part.

DataObjects.Net 5.0 Beta 2

This is the next step in our road to the upcoming major version of DataObjects.Net.

Changes in Beta 2:

• Added support for VB.NET model assemblies
  
• Increased compilation performance
  
• Fixed problem with automatic license activation
  
• Improved logging API
  
• Removed obsolete Transaction.OpenAuto() methods
  
• Session closes connection as soon as the last transaction completes
  
• New Session option NonTransactionalReads.
  
• Added support for Contains operation over keys collection
  
• Added support for persistent fields of DateTimeOffset type
• Key.TypeInfo no longer relies on implicit session
  

Also you may be interested in changes in DataObjects.Net 5.0 Beta 1.

Download

Just as always, releases are available from NuGet gallery.

DataObjects.Net 4.6.5

Changes in DataObjects.Net 4.6.5:

[main] Fixed version validation for entities without explicit version columns
[main] Fixed version validation for decimal version fields
[main] Optimized aggregate translation in complex scenarios
[main] Fixed problem with access to nullable enum columns in final projection
[main] Fixed problem with missing calculated columns in final projection
[main] Fixed materialization of delayed queries with parameters in final projection
[main] Implemented merging of duplicated query parameters
[main] Improved handling of unique index conflicts during persist
[main] Implemented query batching when ValidateEntityVersions option is enabled
[sqlserver] Optimized queries that are executed during driver initialization
[sqlserver] Added support for full text data type columns
[sqlserver] Added ignoring of filetable objects during upgrade and schema validation
[sqlserverce] Fixed translation of DateTime operations in LINQ
[sqlite] Fixed translation of DateTime operations in LINQ
[mysql] Fixed translation of DateTime operations in LINQ

Download

Just as always releases are available at our site and NuGet gallery.

Winter sale

This year we're starting our winter sale right on December 27. Just use the coupon code DO2014 on checkout and get 50% off all editions of DataObjects.Net. If you want to renew your subscription use DO2014RENEWAL to get 70% discount.

This special on our products is valid until Wednesday 15 January 2014.

Should you have any questions, please email to sales@x-tensive.com.

DataObjects.Net 5.0 Beta 1

This is a short announce about the upcoming major version of DataObjects.Net.

Changes:

Added

• PostSharp is replaced with custom persistence weaver based on Mono.Cecil
  
• Support for String.Like extension method with support in LINQ translator
  
• Support for DateTime.ToString("s") in LINQ
  
• Support for ignoring particular tables and/or columns during Domain.Build
  
• Support for defining recycled fields via external definitions
  
• ReSharper annotations for public API
  

Changed

• Xtensive.Tuples performance is significantly improved
  
• Validation framework is reworked
  
• Persist operation is optimized to avoid sorting entities that do not have incoming/outgoing FK
  
• IEnumerable.Remove() extension method is marked as obsolete
  

Removed

• Support for .NET 3.5 and Visual Studio 2008
  
• Object-to-object mapper
  
• Xtensive.Core and Xtensive.Aspects assemblies. Now we have single Xtensive.Orm assembly with a bunch of database drivers in separate assemblies.
  
• Installer. It is no longer provided so use NuGet or binaries package instead
  

Download

Just as always, releases are available from NuGet gallery.

Avoiding connection pool fragmentation

Microsofties admitted that some noticeable performance problems are possible while working with MS SQL Server via ADO.Net (link, scroll to Pool Fragmentation). The connection pooling technique used in ADO.Net to optimize and minimize the cost of opening connections reported to be "not so optimal" in particular scenarios.

The root of the problem is that the exact match between connection string and corresponding connection pool is required. Even if 2 connection strings are build from the same key-value pairs but differ in their order, you get 2 connection pools.

As a result, we get connection pool fragmentation, which is a common problem in many Web applications where the application can create a large number of pools that are not freed until the process exits. This leaves a large number of connections open and consuming memory, which results in poor performance. Surprise for large multi-tenant application developers and maintainers!

From their side, Microsoft is not going to do anything about it in foreseeable future, instead they suggest to use a workaround: connect to master database and open the desired database with a separate command:

// Assumes that command is a SqlCommand object and that
// connectionString connects to master.
command.Text = "USE DatabaseName";
using (SqlConnection connection = new SqlConnection(
  connectionString))
  {
    connection.Open();
    command.ExecuteNonQuery();
  }

How can you achieve the same not leaving the zone of comfortable DataObjects.Net API? Time to get familiar with new member of DomainConfiguration class: ConnectionInitializationSql. The main purpose of the member is to literally execute the provided text as DbCommand right after a connection is opened.

Say, we are using Northwind database:


<domain
name="Default"
provider="sqlserver"
connectionString="Server=myServerAddress;Database=master;..."
connectionInitializationSql="USE NORTHWIND"
...>
<domain>


The feature is available in DataObjects.Net 4.6.4. Download DataObjects.Net

Support for in-memory database

I'm not sure whether anyone remembers, but in the beginning of DataObjects.Net 4.0 epoch we had a semi-successful attempt to provide our own in-memory database implementation, which existed for several years but eventually was removed from the product by various reasons.

And guess what? Starting from DataObjects.Net 4.6.4, in-memory database support is back! But this time it is not our own version, but SQLite in-memory mode.

To connect to in-memory database use the following connection string:

<domain
  name="Default"
  provider="sqlite"
  connectionString="Data Source=:memory:"
  ...>
<domain>

The advantages of such kind of storage are the highest possible speed of database-related operations and no necessity to bother with database files on disk. On the contrary, there are also disadvantages, like the absence of multiple connections to database and no real persistence.

SQLite in-memory mode has some requirements that DataObjects.Net must have met. The most important one is that as soon as connection to in-memory database is closed, the database is destroyed and all data is lost.

Before version 4.6.4, DataObjects.Net always built database scheme in a separate connection which was closed after that procedure. Moreover, the ORM didn't keep any connections open at all, closing them as soon as corresponding session is disposed to free up resources and return connection to connection pool. To support the SQL in-memory mode we had to change the connection management layer and make it more flexible and configurable.

And so we did it. Now we open connection to SQL in-memory database only once on domain build and keep it open until domain is disposed, protecting the database from being destroyed, so your data is safe while domain is alive. In the meantime, you may open and close sessions as usual except you shouldn't open concurrent sessions as only one connection is allowed at a time.

Bad practice: using nested sessions with SQLite in-memory database. Session use the same connection concurrently.

  using (var session1 = domain.OpenSession()) {
    using (var t1 = session1.OpenTransaction()) {

      using (var session2 = domain.OpenSession()) {
        // Here you'll get InvalidOperationException 
        // that the connection is used by another session

Good practice: using subsequent sessions. Both sessions use the same connection, but not concurrently.

  using (var session1 = domain.OpenSession()) {
    using (var t1 = session1.OpenTransaction()) {

      // Do some stuff
      t1.Complete();
    }
  }

  using (var session2 = domain.OpenSession()) {
    using (var t2 = session2.OpenTransaction()) {

      // Do some stuff
      t2.Complete();
    }
  }