Get Dictionary value in IQueryable LINQ to Entities query

Question

I have to support multiple languages in production application.

There are lot of Entity Framework queries that gets data from database as deferred IQueryable list like this:

public IQueryable<Request> GetDeferredRequests()
{
    return _dbContext.Set<Request>();
}   

POCO class looks like this:

public partial class Request
{
    public int RequestID { get; set; }

    public string StatusName { get; set; }

    public string RequestType { get; set; }
}

Data Transfer Object looks like this:

public class RequestDTO
{
    public int RequestID { get; set; }

    public string StatusName { get; set; }

    public string RequestType { get; set; }
}

After that I map the EF POCO entity to Data Transfer Object. To support multiple languages I want to get resource value by database value in mapping like the following method:

public IQueryable<RequestDTO> MapRequests(IQueryable<Request> requests)
{
      Dictionary<string, string> resoures = new Dictionary<string, string>();

      System.Resources.ResourceSet resources = DatabaseResources.ResourceManager.GetResourceSet(new System.Globalization.CultureInfo("en"), true, true);

      foreach (DictionaryEntry resource in resources)
      {
          resoures.Add(resource.Key.ToString(), resource.Value.ToString());
      }

      return requests.Select(c => new RequestDTO()
      {
          RequestID = c.RequestID,
          StatusName =  resoures.Single(r => r.Key == c.StatusName).Value,
          RequestType = resoures.Single(r => r.Key == c.RequestType).Value
      });
}

The problem is that the last command throws the following exception:

LINQ to Entities does not recognize the method 'System.String get_Item(System.String)' method, and this method cannot be translated into a store expression.

Unfortunatelly the converting IQueryable to IEnumerable by ToList() is not an option, because I do not want to move list to memory.

Answer 1

You have to be aware of the difference of IQueryable and IEnumerable.

IEnumerable

An object that implements IEnumerable represents a sequence. It holds everything to get the first element of the sequence and once you've got an element you can get the next element as long as there is an element.

At lowest level, enumerating over this sequence is done by calling GetEnumerator() and repeatedly calling MoveNext(). Every time MoveNext returns true, you've got an element. This element can be accessed using property Current.

Enumerating at this lowest level is seldom done. Usually you enumerate using foreach, or one of the LINQ functions that don't return IEnumerable: ToList(), Count(), Any(), FirstOrDefault(), etc. At the deepest level they all call GetEnumerator and MoveNext / Current.

IQueryable

Although an object that implements IQueryable seems like an IEnumerable, it does not represent the sequence of object itself. It represents the potential to create an IEnumerable sequence.

For this, the IQueryable holds an Expression and a Provider. The Expression is a representation of what data must be queried. The Provider knows who to query for the date (usually a database management system) and what language this DBMS speaks (usually some sort of SQL).

Concatenating IQueryable LINQ statements does not execute the query. It only changes the Expression. To execute the query you need to start enumerating.

Once you start enumerating the IQueryable using GetEnumerator, the Expression is sent to the Provider who will translate the Expression into SQL and execute the query at the DBMS. The returned data is represented as an IEnumerable, of which GetEnumerator is called.

What does this have to do with my question?

The problem is, that the Provider does not know your function MapRequests. Therefore it can't translate it into SQL. In fact even several standard LINQ functions can't be translated into SQL. See Supported and Unsupported LINQ methods.

AsEnumerable

One way to solve this, is to move the selected data to your local process. The local process knows function MapRequests and knows how to Execute it.

Moving data to the local process can be done using ToList(). However, this would be a waste of processing power if after this you will only need a few elements, like Take(3), or FirstOrDefault().

AsEnumerable to the rescue!

Your Provider knows AsEnumerable. It will move the data to your local process. Some dumb providers will do this by fetching all data. Smarter Providers will fetch the data "per page". One page consists a subset of the queried data, for instance only 50 rows. It is still a waste if you only use FirstOrDefault(), but at least you won't have fetched millions of Customers.

It would be nice if you changed MapRequests to an extension method. See Extension Methods Demystified

public static class MyIEnumerableExtensions
{
    public static IEnumerable<RequestDTO> ToRequestDTO( this IEnumerable<Request> requests)
    {
        // your code
        ...
        return requests.Select(request => new RequestDTO
        {
           RequestId = request.RequestId,
           ...
        });
    }

Usage:

IEnumerable<RequestDto> requestDTOs = GetDeferredRequests()

    // only if you don't want all requests:
    .Where(request => ...)

    // move to local process in a smart way:
    AsEnumerable()

    // Convert to RequestDTO:
    .ToRequestDTO();

Note: the query is not executed until your call GetEnumerator() (or foreach, ToList(), Count(), etc). You can even add other IEnumerable functions:

    .Where(requestDTO => requestDTO.StatusName == ...);

Be aware though, that the statements are not executed by the Database Management System, but by your local process.

Can the DBMS map my Requests?

Yet it probably can. You'll have to transport the resources to the database and use simple database functions to convert Request to RequestDTO. If there are many resources in comparison to the number of Requests that you'll have to convert, then it is probably not wise to do. But if for instance you'll have to convert thousands of Request with 100 resources, and after conversion you'll do a Where, or a GroupJoin with another table, it is probably wise to let the DBMS do the conversion.

It seems that every Resource has a Key and a Value.

• StatusName should have the value of the Resource with Key equal to request.StatusName
• RequestType should have the value of the Resource with Key equal to request.RequestType.

So let's rewrite MapRequests into an extension method of IQeryable:

public IQueryable<RequestDTO> ToRequestDto( this IQueryable<Request> requests,
      IEnumerable<KeyValuePair<string, string>> resources)
{
     // TODO: exception if requests == null, resources == null

     return requests.Select(request => new RequestDTO
     {
         RequestId = request.RequestId,

         // from resources, keep only the resource with key equals to StatusName
         // and select the FirstOrDefault value:
         StatusName = resources
                      .Where(resource => resource.Key == request.StatusName)
                      .Select(resource => resource.Value)
                      .FirstOrDefault(),
         // from resources, keep only the resource with key equals to RequestType
         // and select the FirstOrDefault value:
         RequestType = resources
                      .Where(resource => resource.Key == request.RequestType)
                      .Select(resource => resource.Value)
                      .FirstOrDefault(),
     }

Usage:

IEnumerable<KeyValuePair<string, string> resources = ...
var requestDTOs = GetDeferredRequests()
    .Where(request => ...)
    .ToRequestDTO(resources)

    // do other database processing
    .GroupJoin(myOtherTable, ...)
    .Where(...)
    .Take(3);

Now the complete statement will be executed by the Database management system. Most DBMSs are much more optimized to select specific items from a sequence than your process. Besides this looks much neater.

Answer 2

I did some heavy researching and came across a similar but different solution. The problem I was trying to solve is, how can you look up a value from a dictionary with a known primitive key, so that you can use the value within iqueryable and not have to run .AsEnumerable() or .ToList() and pass it back to the app server for slow processing.

All credit to the author here: https://www.mlink.in/qa/?qa=1077916/

First we make the extension class & methods as defined:

/// <summary>
/// Holds extension methods that simplify querying.
/// </summary>
public static class QueryExtensions
{
    /// <summary>
    ///   Return the element that the specified property's value is contained in the specified values.
    /// </summary>
    /// <typeparam name="TElement"> The type of the element. </typeparam>
    /// <typeparam name="TValue"> The type of the values. </typeparam>
    /// <param name="source"> The source. </param>
    /// <param name="propertySelector"> The property to be tested. </param>
    /// <param name="values"> The accepted values of the property. </param>
    /// <returns> The accepted elements. </returns>
    public static IQueryable<TElement> WhereIn<TElement, TValue>(
        this IQueryable<TElement> source, 
        Expression<Func<TElement, TValue>> propertySelector, 
        params TValue[] values)
    {
        return source.Where(GetWhereInExpression(propertySelector, values));
    }

    /// <summary>
    ///   Return the element that the specified property's value is contained in the specified values.
    /// </summary>
    /// <typeparam name="TElement"> The type of the element. </typeparam>
    /// <typeparam name="TValue"> The type of the values. </typeparam>
    /// <param name="source"> The source. </param>
    /// <param name="propertySelector"> The property to be tested. </param>
    /// <param name="values"> The accepted values of the property. </param>
    /// <returns> The accepted elements. </returns>
    public static IQueryable<TElement> WhereIn<TElement, TValue>(
        this IQueryable<TElement> source, 
        Expression<Func<TElement, TValue>> propertySelector, 
        IEnumerable<TValue> values)
    {
        return source.Where(GetWhereInExpression(propertySelector, values.ToList()));
    }

    /// <summary>
    ///   Gets the expression for a "where in" condition.
    /// </summary>
    /// <typeparam name="TElement"> The type of the element. </typeparam>
    /// <typeparam name="TValue"> The type of the value. </typeparam>
    /// <param name="propertySelector"> The property selector. </param>
    /// <param name="values"> The values. </param>
    /// <returns> The expression. </returns>
    private static Expression<Func<TElement, bool>> GetWhereInExpression<TElement, TValue>(
        Expression<Func<TElement, TValue>> propertySelector, ICollection<TValue> values)
    {
        var p = propertySelector.Parameters.Single();
        if (!values.Any())
            return e => false;

        var equals =
            values.Select(
                value =>
                (Expression)Expression.Equal(propertySelector.Body, Expression.Constant(value, typeof(TValue))));
        var body = equals.Aggregate(Expression.OrElse);

        return Expression.Lambda<Func<TElement, bool>>(body, p);
    }
}

then, we can use it to query where value in dictionary as follows:

            var countryCitiesLookup = Dictionary<string, List<string>>();
            ///
            // fill in values here
            ///
            var country = "USA";
            var x = _context.Countries.Where(x => true)// original query without 
                .WhereIn(x => x.CountryID, CountryCitiesLookup.Keys.Contains(country));`enter code here`