puh... after many hours i found a solution:

Code: 
[DllImport("DLMF.dll", EntryPoint = "RSDFToPlain", ExactSpelling = true, CallingConvention = CallingConvention.StdCall, CharSet = CharSet.Ansi)]
public static extern void RSDFToPlain([MarshalAs(UnmanagedType.BStr)]string RSDFFile, [MarshalAs(UnmanagedType.BStr)]out string Result);

            string _file = Environment.CurrentDirectory + "\\a.rsdf";
            string _data;

            RSDFToPlain(_file, out _data);
            Console.WriteLine(_data);
            Console.ReadKey(true);
like in c++ its not possible to give a string as return value.. so i needed to change this

EDIT:
btw. you need to download the dlmf.zip again, i've updated the files etc. sorry for all these problems, code is tested and works now