How to handle non-persistent collection

Today we've faced with an interesting scenario from Jensen: an Entity has a non-persistent collection which should be serialized to and deserialized from a persistent field on demand. Here is how the problem was initially described on the support website (go to the end of the page):

Object A should contain a list of samples (object T). I could do something like

A a = GetA();
a.Load(); (convert the byte[] to a List)

and something like

a.Store(); (convert the List to a byte[])
m_transactionScope.Commit();

But I don't like calling a Load() and Store() function each time, it would be easier if this could be automated. When A is loaded from the database it should Load(), when saved to the database it should Store().

After several experiments I've found a solution.

1. Add a set of properties to the desired class.

public class MyEntity : Entity {
    ...
    [Field] // Is used as an indicator that collection has been changed
    private bool IsChanged { get; set; }

    [Field] // Backing field for serialized collection value
    private byte[] CollectionValue { get; set; }

    // The collection itself. Note that it is not persistent.
    public ObservableCollection<int> Collection { get; set; }
    ...

2. Add proper methods for serialization & deserialization:

    private static byte[] Serialize(ObservableCollection<int> collection)
    {
      var ms = new MemoryStream();
      var bf = new BinaryFormatter();
      bf.Serialize(ms, collection);
      return ms.ToArray();
    }

    private static ObservableCollection<int> Deserialize(byte[] bytes)
    {
      var bf = new BinaryFormatter();
      var ms = new MemoryStream(bytes);
      return (ObservableCollection<int>) bf.Deserialize(ms);
    }

3. Add Entity initializer & event handlers:

    protected override void OnInitialize()
    {
      base.OnInitialize();
      // Initializing the collection
      if (CollectionValue != null && CollectionValue.Length > 0)
        Collection = Deserialize(CollectionValue);
      else
        Collection = new ObservableCollection<int>();

      // Subscribing to events
      Collection.CollectionChanged += Collection_CollectionChanged;
      Session.Events.Persisting += Session_Persisting;
    }

    void Session_Persisting(object sender, EventArgs e)
    {
      if (!IsChanged)
        return;

      // Serializing the collection right before persist, if it is changed
      CollectionValue = Serialize(Collection);
      IsChanged = false;
    }

    void Collection_CollectionChanged(object sender, NotifyCollectionChangedEventArgs e)
    {
      IsChanged = true;
    }

So, what is going on here?
An instance of MyEntity is subscribed to a CollectionChanged event of MyEntity.Collection and updates MyEntity.IsChanged property when CollectionChanged event is fired. As the property is persistent, the MyEntity instance is added into a queue for persisting. Right before the persist procedure takes place, the instance of MyEntity receives a notification from its Session. During handling of this event we check whether the collection has been changed or not. If so, we update MyEntity.CollectionValue property and changes are successfully persisted to a database.

Note, that I've decided to use ObservableCollection type instead of List because the solution requires a notifications from the collection.

Hope this helps.

Making connection strings secure

Recently, Paul Sinnema from Diartis AG asked on the support website how to make connection strings, listed in application configuration files (app.config or web.config), more secure.

For that time the only thing we could suggest was to use a workaround like that:

1. Encrypt the required connection string with encryption method you prefer and put it somewhere in web.config/app.config file in encrypted form.

2. Load DomainConfiguration through standard API:

var config = DomainConfiguration.Load("mydomain");

3. Set manually config.ConnectionInfo property with decrypted connection string

config.ConnectionInfo = new ConnectionInfo("sqlserver",
  DecryptMyConnectionString(encryptedConnectionString));

4. Build Domain with the config.

Paul answered that although the workaround is useful, he'd prefer that DataObjects.Net could be integrated somehow with standard .NET configuration subsystem that provides easy encrypting/decrypting of connection strings.

Let me explain the approach, Paul was talking about. For example, if plain connection string configuration looks like that;

  <connectionStrings>
     <add name="cn1" connectionString="Data Source=localhost\SQL2008;Initial Catalog=DO40-Tests;Integrated Security=True;MultipleActiveResultSets=True" />
  </connectionStrings>

then after executing "aspnet_regiis.exe -pef "connectionStrings" %PATH_TO_PROJECT%" the encrypted one will look like the following:

  <connectionStrings configProtectionProvider="RsaProtectedConfigurationProvider">
    <EncryptedData Type="http://www.w3.org/2001/04/xmlenc#Element"
      xmlns="http://www.w3.org/2001/04/xmlenc#">
      <EncryptionMethod Algorithm="http://www.w3.org/2001/04/xmlenc#tripledes-cbc" />
      <KeyInfo xmlns="http://www.w3.org/2000/09/xmldsig#">
        <EncryptedKey xmlns="http://www.w3.org/2001/04/xmlenc#">
          <EncryptionMethod Algorithm="http://www.w3.org/2001/04/xmlenc#rsa-1_5" />
          <KeyInfo xmlns="http://www.w3.org/2000/09/xmldsig#">
            <KeyName>Rsa Key</KeyName>
          </KeyInfo>
          <CipherData>
            <CipherValue>TJETPde0GVrSAKk0VpUs3EsV8XFRQrVU1lGlrX8aEhvfa4MaMpN3O6KLo3I9e5oujXBImKtmXHWxnF6KRcSWteuQXD4I37Dq6yE1cZ4qDHqNMkcQK+lgi6jbRLcPzYQdFuOLagsAVpif8OvY0dJkiKd+Srqjwz7ZO225VCipaQs=</CipherValue>
          </CipherData>
        </EncryptedKey>
      </KeyInfo>
      <CipherData>
        <CipherValue>6bQyCUdH9rqdNAD4IwmSXtAQIEuRfSjpQ/Py5ovz2ZpwJdEXa4W3OKWMm52g1ARXPqMYw1thfQae//+1JWTAPQwrE3JTZEdcgu47KE2x8uqdWeJfoXSecp6berSPc4qAa4F2B5x8weUGPiAsBldVfBXwUBPniKWbnnXViJWmcSn16kaAtIkq5GtNOp7YhPt9WKZG0QkFRpEaRAWqLgxK6nXUwK8BgbGEBI4B27QSFj2ME1/JI+o//RrchntEuW6VolEA7PAejMujsIn1xb0hqg==</CipherValue>
      </CipherData>
    </EncryptedData>
  </connectionStrings>

As a result, the connection strings get encrypted and therefore, much more secure. Working with them is as simple as usual: they can be accesses through standard ConfigurationManager without any special tricks:

ConfigurationManager.ConnectionStrings["cn1"].ConnectionString

And this simplicity makes the whole approach extremely useful.

So, what could be done to get DataObjects.Net compatible with all this shiny stuff, keeping in mind that connection strings section is outside DataObjects.Net configuration?

After thinking for a while on the problem, we've got the following idea: what if we put in domain configuration not a connection string itself, but a reference to the connection string, that is actually defined in connectionStrings section? For instance:

<Xtensive.Orm>
    <domains>
      <domain provider="sqlserver" connectionString="#cn1" ... />
    </domain>
  </domains>
<Xtensive.Orm>

Where:

• cn1 is a name of a connection string defined in connectionStrings section;
• # sign is used to inform that actually this is a reference to a connection string and DataObjects.Net must resolve it through ConfigurationManager.

Having this feature implemented, DataObjects.Net will provide support for encrypted connection strings out-of-the-box.

And now, we'd like to ask you some important questions:

• What would you say about the feature?
• Will it be useful for you and should it be implemented?
• Is the "#name" notation for setting a reference to a connection string is more or less clear or wouldn't it better to change it to something else?

Dear users of DataObjects.Net, we need your feedback. Please, post a comment or two.

Thanks in advance.

Alex Yakunin's blog:

• Simple audit logging with DO4: code + some analysis of DO4 brains :)
• Default logging mode in DO4 is changed
• Thoughts: what must be done ASAP to improve v.4.2?
• Link: "Volatile keyword in C# – memory model explained"
• DataObjects.Net v4.2 is released!
• Help Server v3.0.1 is released!
• News & links to new topics in furum
• Alternative DataObjects.Net download location
• Versions and locking tests (code samples for Manual): the links
• DataObjects.Net v4.2 RC2 is available
• My Google Buzz feed
• X-tensive.com is updated (warning: beta version!)
• DataObjects.Net v4.2 RC is available
• Test pass results
• Support for connection strings and new way of setting default schema
• I'm back ;)
• The purpose of ProxyKeyGenerator
• Quest of the day
• New chapter in Manual: "Hacker's guide to DataObjects.Net"
• One more Manual update
• Why there is no [DebuggerSkipThrownException(typeof(...))]?
• Preliminary version of DataObjects.Net Manual in PDF
• DisconnectedState, OperationSet, events and operation logging
• Happy New Year!
• LINQ is nice: the same problem solved by different LINQ queries
• Testing new version of Help Server

Dmitri Maximov's blog:

• Denis Krjuchkov vs. Oracle
• DataObjects.Net 4.2 RC’s health status
• Google URL shortener for everyone
• Microsoft Sql Server CE 3.5 provider for DataObjects.Net 4
• File sync problem and possible solutions
• A code quality assessment concept
• Ayende on Dependency Injection
• Updates on the development process
• A frank confession
• Query preprocessors, Inversion of control & Localization support LINQ translator extension
• Logging, part 3. Configuring logging through log4net
• Logging, part 2. Architecture & configuration
• Logging, part 1. Introduction

Alex Kofman's blog:

• Persistent Hierarchies and Key Generators
• New WPF sample
• WPF applications and DataObjects.Net

Denis Krjuchkov's blog (new!):

• LINQ
• Unified storage level exceptions
• PostSharp is going closed source
• Practice of functional programming
• Oracle challenge
• Yep, I'm here

Alex Ilyin's blog (new!):

• WPF Resources
• CSLA – First impression
• Code quality test
• Inventing Log module
• Web forms and ASP.NET MVC
• Between Tirany and Democracy

P.S. Guys, sorry, but I just decided to share your Skype pics here - people should know your real nature :)

// Actually I did this just to add some colors. And I know they don't read comments.

Thoughts: being first vs being best

I just read a nice post from DevExpress CTO about this.

We've been among first DO1.X, 2.X and 3.X - there were tons of unique features at that moment.

Some examples:
- We've been using runtime proxies initially - starting from v1.0 in 2003. NHibernate started to use them only in 2005. Now we're using PostSharp-based aspects, and I suspect, many others already started to look onto this ;)
- We've been first who integrated full-text search & indexing into ORM. Initial version supporting Microsoft Search service appeared in 2003; Lucene.Net support have been added in 2005. Now this feature is supported by many other ORM frameworks - e.g. NHibernate and Lightspeed.
- The same is about translatable properties - we've been supporting them starting from 2003. Now they're implemented e.g. in Genom-e.
- We've implemented versioning extension in 2005. Genom-e team added similar historization feature just recently.
- And finally, schema upgrade - it was a part of DO starting from v1.0. Now it exists in many frameworks, but is always implemented as design-time feature. Btw, I understand this brings some benefits (btw, we're going to provide similar feature shortly), but what about runtime? As application users, we used to install new versions of applications without caring about running upgrade scripts. Moreover, as developer, I'd prefer getting all necessary upgrade logic executed automatically. So why all these frameworks are capable only of generating such scripts at design time? Who'll combine them? Who will execute them? Who will produce their versions for other RDBMS? Who will check if existing database version isn't too old? I think, this approach also leads quite many TODOs for very common tasks to leave them for developers.

And, AFAIK, the following features are still unique:
- Paired (inverse) properties - at least as this is done in DO. I.e. in fully commutative way.
- Persistent interfaces - again, I'm speaking about our own version of this feature.
- Access control \ security system
- Action-level instead of state level change logging (used for disconnected state change replication).

So why did DO3.X die? Because of architectural lacks, that initially allowed us to develop it fast. E.g. our mapping support was quite limited. Really, being first usually != being best.

Do you know, that:
- I was 23 years old when first version of DO has been released. I'm 10 years younger than e.g. Frans Bouma ;)
- I had a good experience with RDBMS at that moment, but as you may find, my experience was mainly limited by the scope of SMB web applications, and finally this lead to some architectural lacks in v1.X.
- I had really good programming background - my CV was very good at that point. But I didn't develop a framework comparable by the scale to DO before. But framework development guidelines are quite different in comparison to application development guidelines. Initial architectural lacks are much more painful here: in some cases you simply can't overwhelm them by adding N-th module.

On the other hand, we've got huge ORM and database experience with v1.X-3.X. We've became real experts. We've studied & fixed lots of cases and issues - starting from very frequent ones to those that seem quite hard to face in a particular application (but this doesn't mean you shouldn't keep them in mind). We know much more about what is important, and how this must work.

And what's more important, we've been ultimately the first ones exploring many new paths and features - take a look at above lists ;) Most of features there were later adopted by others - this proves they were good enough, and what's more important, this shows we can generate and implement such ideas earlier than others do. May be that's because I don't like to barely repeat the others. Take NHibernate as an example: isn't it really boring to follow the path passed by others few years ago (I mean hibernate)? Ok, it is already successful, and such a path offers attractive way to start. But is it a good enough reason to plainly repeat it, instead of making something better?

Ok, it was really a kind of rush for us, until we reached our own limits - being the first. But as you know, we didn't stopped on this! We just started a new rush - I hope this shows well what kind characters are staying behind our team ;) Moreover, all these years we've been growing up using all the opportunities we had, and this allowed us to accomplish our almost unimaginable complex task at all. Yes, now we're the only ones supporting not just third-party databases, but having our own, shiny-new, real RDBMS integrated with ORM. And we're almost ready to show the full power of this competitive advantage (wait for sync, and further - Mono Silverlight support). We're going to eliminate necessity to study and use the whole spectra of technologies you need in most of cases, including Entity Framework, Sync Framework, ADO.NET Data Services, .NET RIA Services, SQL Server Compact \ SQLite - and this isn't a full list ;)

Btw, probably, we're the only ORM vendors that won't suffer much from EF appearance: first of all, because our paths are probably the most distant ones from the point of approach to the problem. And secondly, because we already suffered enough from 2-year pause in releases ;) I feel it will be the hard time for many many others - especially the ones offering similar features. LLBLGen Pro, Genom-e, Subsonic, Lightspeed, etc., and even NHibernate (still no LINQ!) - guys, are you well prepared for this fight? At least we are - you know, Russians are used to win the wars during long and cold winters ;)

So what about being first vs being best?

It's simple. We've been carefully taking our time for 2 years. If DO1.X-3.X were mainly the first, we're making DO4.0 mainly the best - although from many points of views it is the first one as well. I'm quite happy that the most complex part of our development path is already passed now. The wheel is rotating now, its speed is growing up. We delivered v4.0 + 2 updates just in June. July promises to be even more attractive from the point of features we're going to deliver.

So I'm repeating myself once more: join DO4 camp ;)