Parsing arbitrary FIX messages in Scala

When to engage the type system


Thomas P. Harte

Saturday, 2025-04-12 (University of Illinois at Chicago)

Disclaimer

Disclaimer

Thomas P. Harte ("the Author") is providing this presentation and its contents ("the Content") for educational purposes only at the Open Source Quantitative Finance conference, Chicago, IL (2025-04-12). The Author is not a registered investment advisor and the Author does not purport to offer investment advice nor business advice. The opinions expressed in the Content are solely those of the Author, and do not necessarily represent the opinions of the Author's employer, nor any organization, committee or other group with which the Author is affiliated.

THE AUTHOR SPECIFICALLY DISCLAIMS ANY PERSONAL LIABILITY, LOSS OR RISK INCURRED AS A CONSEQUENCE OF THE USE AND APPLICATION, EITHER DIRECTLY OR INDIRECTLY, OF THE CONTENT. THE AUTHOR SPECIFICALLY DISCLAIMS ANY REPRESENTATION, WHETHER EXPLICIT OR IMPLIED, THAT APPLYING THE CONTENT WILL LEAD TO SIMILAR RESULTS IN A BUSINESS SETTING. THE RESULTS PRESENTED IN THE CONTENT ARE NOT NECESSARILY TYPICAL AND SHOULD NOT DETERMINE EXPECTATIONS OF FINANCIAL OR BUSINESS RESULTS.

Motivation

FIX

fix-wikipedia.png

Source: https://en.wikipedia.org/wiki/Financial_Information_eXchange

Functional programming

  • hardware has changed
    • multi-core processors
    • clusters
  • computer languages have changed
    • Turing: imperative
    • Church: functional
  • functional programming languages
    • immutability of state ("no shared mutable state")
    • pure functions / referentially transparent ("no side effects")
    • higher-order functions ("first-class objects")
    • recursion ("no loops")
  • end game: scalability

Functional languages

haskell.png

scala.png

Scala's killer app

spark.png

  • Unified engine for large-scale data analytics
    • single-node machines
    • clusters
  • API supports multiple languages
    • Scala
    • Java
    • R
    • Python
    • Spark SQL

spark-download.png 

bash$ version=spark-3.5.5-bin-hadoop3-scala2.13
bash$ url=https://dlcdn.apache.org/spark/spark-3.5.5
bash$ curl --compressed $url/$version.tgz > $version && tar xvfpz $version 

% Total    % Received % Xferd  Average Speed   Time    Time     Time  Current                                                         
                               Dload  Upload   Total   Spent    Left  Speed                                                           
100  390M  100  390M    0     0  88.5M      0  0:00:04  0:00:04 --:--:-- 89.4M                                                          
spark-3.5.5-bin-hadoop3-scala2.13/                                                                                                      
spark-3.5.5-bin-hadoop3-scala2.13/jars/                                                                                                 
spark-3.5.5-bin-hadoop3-scala2.13/jars/HikariCP-2.5.1.jar                                                                               
spark-3.5.5-bin-hadoop3-scala2.13/jars/JLargeArrays-1.5.jar                                                                             
spark-3.5.5-bin-hadoop3-scala2.13/jars/JTransforms-3.1.jar                                                                              
spark-3.5.5-bin-hadoop3-scala2.13/jars/RoaringBitmap-0.9.45.jar    

[snip, snip]

spark-3.5.5-bin-hadoop3-scala2.13/R/lib/SparkR/help/aliases.rds
spark-3.5.5-bin-hadoop3-scala2.13/R/lib/SparkR/html/
spark-3.5.5-bin-hadoop3-scala2.13/R/lib/SparkR/html/00Index.html
spark-3.5.5-bin-hadoop3-scala2.13/R/lib/SparkR/html/R.css
spark-3.5.5-bin-hadoop3-scala2.13/R/lib/SparkR/INDEX
spark-3.5.5-bin-hadoop3-scala2.13/R/lib/sparkr.zip


bash$ $version/bin/spark-shell

spark-shell.png

FIX

FIX is all about XML

Snag some FIXML (it's not "real" FIX)

fixml.png

Source: FIX Antenna .NET Programmer's Guide 

import scala.xml.*

val fixml =
 <FIXML>
   <Order ID="123456" Side="2" TxnTm="2001-09-11T09:30:47-05:00" Typ="2" Px="93.25" Acct="26522154">
     <Hdr Snt="2001-09-11T09:30:47-05:00" PosDup="N" PosRsnd="N" SeqNum="521" SID="AFUNDMGR" TID="ABROKER"/>
     <Instrmt Sym="IBM" ID="459200101" Src="1"/>
     <OrdQty Qty="1000"/>
   </Order>
</FIXML>

Work the FIXML

import scala.xml.*

val fixml =
    <FIXML>
    <Order ID="123456" Side="2" TxnTm="2001-09-11T09:30:47-05:00" Typ="2" Px="93.25" Acct="26522154">
        <Hdr Snt="2001-09-11T09:30:47-05:00" PosDup="N" PosRsnd="N" SeqNum="521" SID="AFUNDMGR" TID="ABROKER"/>
        <Instrmt Sym="IBM" ID="459200101" Src="1"/>
        <OrdQty Qty="1000"/>
    </Order>
    <Order ID="456789" Side="1" TxnTm="2001-09-11T09:30:47-05:00" Typ="2" Px="100.25" Acct="15426522">
        <Hdr Snt="2001-09-11T09:30:47-05:00" PosDup="N" PosRsnd="N" SeqNum="522" SID="FUNDMGR-1" TID="BROKERAB"/>
        <Instrmt Sym="TSLA" ID="459200101" Src="1"/>
        <OrdQty Qty="100"/>
    </Order>
    </FIXML>


(fixml \ "Order").map(_ \@ "TxnTm")
(fixml \ "Order").map(_ \ "Hdr" \@ "Snt")


res4_0: Seq[String] = List("2001-09-11T09:30:47-05:00", "2001-09-11T09:30:47-05:00")
res4_1: Seq[String] = List("2001-09-11T09:30:47-05:00", "2001-09-11T09:30:47-05:00")

Work the FIXML

import scala.xml.*

val fixml =
    <FIXML>
    <Order ID="123456" Side="2" TxnTm="2001-09-11T09:30:47-05:00" Typ="2" Px="93.25" Acct="26522154">
        <Hdr Snt="2001-09-11T09:30:47-05:00" PosDup="N" PosRsnd="N" SeqNum="521" SID="AFUNDMGR" TID="ABROKER"/>
        <Instrmt Sym="IBM" ID="459200101" Src="1"/>
        <OrdQty Qty="1000"/>
    </Order>
    <Order ID="456789" Side="1" TxnTm="2001-09-11T09:30:47-05:00" Typ="2" Px="100.25" Acct="15426522">
        <Hdr Snt="2001-09-11T09:30:47-05:00" PosDup="N" PosRsnd="N" SeqNum="522" SID="FUNDMGR-1" TID="BROKERAB"/>
        <Instrmt Sym="TSLA" ID="459200101" Src="1"/>
        <OrdQty Qty="100"/>
    </Order>
    </FIXML>


val orders = fixml \ "Order"
val hdr = orders.map(_ \ "Hdr")
orders.map(_ \@ "TxnTm")
hdr.map(_ \@ "Snt")


res6_2: Seq[String] = List("2001-09-11T09:30:47-05:00", "2001-09-11T09:30:47-05:00")
res6_3: Seq[String] = List("2001-09-11T09:30:47-05:00", "2001-09-11T09:30:47-05:00")

Work the FIXML

import scala.xml.*

val fixml =
    <FIXML>
    <Order ID="123456" Side="2" TxnTm="2001-09-11T09:30:47-05:00" Typ="2" Px="93.25" Acct="26522154">
        <Hdr Snt="2001-09-11T09:30:47-05:00" PosDup="N" PosRsnd="N" SeqNum="521" SID="AFUNDMGR" TID="ABROKER"/>
        <Instrmt Sym="IBM" ID="459200101" Src="1"/>
        <OrdQty Qty="1000"/>
    </Order>
    <Order ID="456789" Side="1" TxnTm="2001-09-11T09:30:47-05:00" Typ="2" Px="100.25" Acct="15426522">
        <Hdr Snt="2001-09-11T09:30:47-05:00" PosDup="N" PosRsnd="N" SeqNum="522" SID="FUNDMGR-1" TID="BROKERAB"/>
        <Instrmt Sym="TSLA" ID="459200101" Src="1"/>
        <OrdQty Qty="100"/>
    </Order>
    </FIXML>


for {
    order <- fixml \ "Order"
    hdr <- order \ "Hdr"
} yield (order \@ "TxnTm", hdr \@ "Snt")


res11: Seq[(String, String)] = List(
  ("2001-09-11T09:30:47-05:00", "2001-09-11T09:30:47-05:00"),
  ("2001-09-11T09:30:47-05:00", "2001-09-11T09:30:47-05:00")
)

FIX doesn't look like FIXML

Snag some "real" FIX

real-fix.png

Source: FIX Antenna .NET Programmer's Guide 

val fix = """
8=FIX.4.2<SOH>9=153<SOH>35=D<SOH>49=BLP<SOH>56=SCHB<SOH>34=1<SOH>50=30737<SOH>97=Y<SOH>
52=20000809-20:20:50<SOH>11=90001008<SOH>1=10030003<SOH>21=2<SOH>55=TESTA<SOH>54=1<SOH>38=4000<SOH>
40=2<SOH>59=0<SOH>44=30<SOH>47=I<SOH>60=20000809-18:20:32<SOH>10=061<SOH>
""".replace("\n", "").replace("<SOH>", "|")


fix: String = "8=FIX.4.2|9=153|35=D|49=BLP|56=SCHB|34=1|50=30737|97=Y|52=20000809-20:20:50|11=90001008|1=10030003|21=2|55=TESTA|54=1|38=4000|40=2|59=0|44=30|47=I|60=20000809-18:20:32|10=061|"

Work the "real" FIX


fix: String = "8=FIX.4.2|9=153|35=D|49=BLP|56=SCHB|34=1|50=30737|97=Y|52=20000809-20:20:50|11=90001008|1=10030003|21=2|55=TESTA|54=1|38=4000|40=2|59=0|44=30|47=I|60=20000809-18:20:32|10=061|"


fix.split('|')
fix.split('|').filter(x => x.startsWith("60=") | x.startsWith("52=")).map(_.substring(3))

res25_1: Array[String] = Array(
  "8=FIX.4.2",
  "9=153",
  "35=D",
  "49=BLP",
  "56=SCHB",
  "34=1",
  "50=30737",
  "97=Y",
  "52=20000809-20:20:50",
  "11=90001008",
  "1=10030003",
  "21=2",
  "55=TESTA",
  "54=1",
  "38=4000",
  "40=2",
  "59=0",
  "44=30",
  "47=I",
  "60=20000809-18:20:32",
  "10=061"
)
res25_2: Array[String] = Array("20000809-20:20:50", "20000809-18:20:32")

FIX Specs

"Canonical" reference

fix-tag-60.png

Source: FIXimate: FIX Interactive Message And Tag Explorer

"Canonical" reference

fix-tag-52.png

Source: FIXimate: FIX Interactive Message And Tag Explorer

Actual reference (XML again)

fix-xml-spec-50sp2.png

Source: quickfix/spec/FIX50SP2.xml

Anatomy of an XML FIX spec

fix50sp2-gross-anatomy-1.png

Source: quickfix/spec/FIX50SP2.xml

Anatomy of an XML FIX spec

fix50sp2-gross-anatomy-2.png

Source: quickfix/spec/FIX50SP2.xml

Anatomy of an XML FIX spec

fix50sp2-gross-anatomy-tag-60.png

Source: quickfix/spec/FIX50SP2.xml

Anatomy of an XML FIX spec

fix50sp2-gross-anatomy-tag-52.png

Source: quickfix/spec/FIX50SP2.xml

Gross Anatomy of an XML FIX spec

gross-anatomy-summary-table.png

Work the "real" FIX (fields only)


fix: String = "8=FIX.4.2|9=153|35=D|49=BLP|56=SCHB|34=1|50=30737|97=Y|52=20000809-20:20:50|11=90001008|1=10030003|21=2|55=TESTA|54=1|38=4000|40=2|59=0|44=30|47=I|60=20000809-18:20:32|10=061|"


fix.split('|')
fix.split('|').filter(x => x.startsWith("60=") | x.startsWith("52=")).map(_.substring(3))

res25_1: Array[String] = Array(
  "8=FIX.4.2",
  "9=153",
  "35=D",
  "49=BLP",
  "56=SCHB",
  "34=1",
  "50=30737",
  "97=Y",
  "52=20000809-20:20:50",
  "11=90001008",
  "1=10030003",
  "21=2",
  "55=TESTA",
  "54=1",
  "38=4000",
  "40=2",
  "59=0",
  "44=30",
  "47=I",
  "60=20000809-18:20:32",
  "10=061"
)
res25_2: Array[String] = Array("20000809-20:20:50", "20000809-18:20:32")

Map tags to field definitions

Mapping tags to field definitions

Read FIX 4.2

Read FIX 4.2 spec

def get_xml_specs() = {
    Seq(
        "FIX40.xml",
        "FIX41.xml",
        "FIX42.xml",
        "FIX43.xml",
        "FIX44.xml",
        "FIX50.xml",
        "FIXT11.xml",
        "FIX50SP1.xml",
        "FIX50SP2.xml",
    )
}

def get_xml_filename(name:String="FIX50SP2.xml") = {
    assert(get_xml_specs().filter(_ == name).length != 0)
    val folder = os.pwd/"fix-specs"
    val filename = folder / name

    filename
}

def get_xml(name:String="FIX50SP2.xml") = {
    XML.loadFile(get_xml_filename(name).toString)
}

Read FIX 4.2 spec

get_xml("FIX42.xml")

res36: Elem = <fix type="FIX" major="4" minor="2" servicepack="0">
 <header>
  <field name="BeginString" required="Y"/>
  <field name="BodyLength" required="Y"/>
  <field name="MsgType" required="Y"/>
  <field name="SenderCompID" required="Y"/>
  <field name="TargetCompID" required="Y"/>
  <field name="OnBehalfOfCompID" required="N"/>
  <field name="DeliverToCompID" required="N"/>
  <field name="SecureDataLen" required="N"/>
  <field name="SecureData" required="N"/>
  <field name="MsgSeqNum" required="Y"/>
  <field name="SenderSubID" required="N"/>
  <field name="SenderLocationID" required="N"/>
  <field name="TargetSubID" required="N"/>
  <field name="TargetLocationID" required="N"/>
  <field name="OnBehalfOfSubID" required="N"/>
  <field name="OnBehalfOfLocationID" required="N"/>
  <field name="DeliverToSubID" required="N"/>
  <field name="DeliverToLocationID" required="N"/>
  <field name="PossDupFlag" required="N"/>
  <field name="PossResend" required="N"/>
  <field name="SendingTime" required="Y"/>
  <field name="OrigSendingTime" required="N"/>
  <field name="XmlDataLen" required="N"/>
  <field name="XmlData" required="N"/>
  <field name="MessageEncoding" required="N"/>
  <field name="LastMsgSeqNumProcessed" required="N"/>
  <field name="OnBehalfOfSendingTime" required="N"/>
...

FIX 4.2: fields

Gather fields

val xml = get_xml("FIX42.xml")
val fields = xml \ "fields" \ "field"

fields: NodeSeq = Seq(
  <field number="1" name="Account" type="STRING"/>,
  <field number="2" name="AdvId" type="STRING"/>,
  <field number="3" name="AdvRefID" type="STRING"/>,
  <field number="4" name="AdvSide" type="CHAR">
   <value enum="B" description="BUY"/>
   <value enum="S" description="SELL"/>
   <value enum="T" description="TRADE"/>
   <value enum="X" description="CROSS"/>
  </field>,
  <field number="5" name="AdvTransType" type="STRING">
   <value enum="C" description="CANCEL"/>
   <value enum="N" description="NEW"/>
   <value enum="R" description="REPLACE"/>
  </field>,
  <field number="6" name="AvgPx" type="PRICE"/>,
  <field number="7" name="BeginSeqNo" type="INT"/>,
  <field number="8" name="BeginString" type="STRING"/>,
  <field number="9" name="BodyLength" type="INT"/>,
  <field number="10" name="CheckSum" type="STRING"/>,
  <field number="11" name="ClOrdID" type="STRING"/>,
  <field number="12" name="Commission" type="AMT"/>,
  <field number="13" name="CommType" type="CHAR">
   <value enum="1" description="PER_UNIT"/>
   <value enum="2" description="PERCENT"/>
   <value enum="3" description="ABSOLUTE"/>
  </field>,
  <field number="14" name="CumQty" type="QTY"/>,
  <field number="15" name="Currency" type="CURRENCY"/>,
...

Map methods

val fieldTagMap = (() => {
    fields.
        map(x => (x \@ "number", x \@ "name", x \@ "type")).
        map(x => (x._1.toInt -> (x._1.toInt, x._2, x._3))).
        toMap
})()

val fieldNameMap = (() => {
    fields.
        map(x => (x \@ "number", x \@ "name", x \@ "type")).
        map(x => (x._2 -> (x._1.toInt, x._2, x._3))).
        toMap
})()

Tag (Int) to field

fieldTagMap(60)
fieldTagMap(52)

res48: (Int, String, String) = (60, "TransactTime", "UTCTIMESTAMP")
res49: (Int, String, String) = (52, "SendingTime", "UTCTIMESTAMP")

Field name (String) to field

fieldNameMap("TransactTime")
fieldNameMap("SendingTime")

res51: (Int, String, String) = (60, "TransactTime", "UTCTIMESTAMP")
res52: (Int, String, String) = (52, "SendingTime", "UTCTIMESTAMP")

Cache the fields

val field_cache = scala.collection.mutable.Map[String,Field]()

def get_field(number:String) =
    field_cache.getOrElseUpdate(number, Field.apply(number, fields))

case class Field(
    number:String,
    name:String,
    fixType:String,
    value:Map[String,String]
)

object Field {
    def apply(number:String, fields:NodeSeq):Field = {
        val field = fields.filter(x => x \@ "number" == number)

        val empty_value = Map[String,String]()
        val empty_field = Field(number, "[N/A]", "[N/A]", empty_value)
        if (field.length == 0) return(empty_field)

        val name = field \@ "name" 
        val fixType = field \@ "type"

        val value = (() => {
            val value = field \\ "value"
            if (value.length==0) {
                empty_value 
            }
            else {
                value.map(x => (x \@ "enum", x \@ "description")).toMap 
            }
        }:Map[String,String])()

        Field(number, name, fixType, value)
    }
}

Retrieve Field from the cache

get_field("60")
get_field("52")

res72: Field = Field(number = "60", name = "TransactTime", fixType = "UTCTIMESTAMP", value = Map())
res73: Field = Field(number = "52", name = "SendingTime", fixType = "UTCTIMESTAMP", value = Map())

Retrieve Field from the cache

get_field("35")

res74: Field = Field(
  number = "35",
  name = "MsgType",
  fixType = "STRING",
  value = HashMap(
    "8" -> "EXECUTION_REPORT",
    "4" -> "SEQUENCE_RESET",
    "f" -> "SECURITY_STATUS",
    "F" -> "ORDER_CANCEL_REQUEST",
    "5" -> "LOGOUT",
    "i" -> "MASS_QUOTE",
    "P" -> "ALLOCATION_INSTRUCTION_ACK",
    "0" -> "HEARTBEAT",
    "H" -> "ORDER_STATUS_REQUEST",
    "7" -> "ADVERTISEMENT",
    "3" -> "REJECT",
    "k" -> "BID_REQUEST",
    "D" -> "NEW_ORDER_SINGLE",
    "E" -> "NEW_ORDER_LIST",
    "e" -> "SECURITY_STATUS_REQUEST",
    "X" -> "MARKET_DATA_INCREMENTAL_REFRESH",
    "9" -> "ORDER_CANCEL_REJECT",
    "N" -> "LIST_STATUS",
    "j" -> "BUSINESS_MESSAGE_REJECT",
    "T" -> "SETTLEMENT_INSTRUCTIONS",
    "Y" -> "MARKET_DATA_REQUEST_REJECT",
    "J" -> "ALLOCATION_INSTRUCTION",
    "A" -> "LOGON",
    "a" -> "QUOTE_STATUS_REQUEST",
...

Parse FIX

Parse a FIX message

def parse(msg:String, sep:Char='|') = {
    val tokens = msg.split(sep)

    tokens.map((x) => {
        val s = x.split('=')
        val tag = s(0); val value = s(1)
        val field = get_field(tag)
        val description = field.value.getOrElse(value, "")

        (tag, field.name, value, description)
    })
}

Parse a FIX message


fix: String = "8=FIX.4.2|9=153|35=D|49=BLP|56=SCHB|34=1|50=30737|97=Y|52=20000809-20:20:50|11=90001008|1=10030003|21=2|55=TESTA|54=1|38=4000|40=2|59=0|44=30|47=I|60=20000809-18:20:32|10=061|"


parse(fix)

res79: Array[(String, String, String, String)] = Array(
  ("8", "BeginString", "FIX.4.2", ""),
  ("9", "BodyLength", "153", ""),
  ("35", "MsgType", "D", "NEW_ORDER_SINGLE"),
  ("49", "SenderCompID", "BLP", ""),
  ("56", "TargetCompID", "SCHB", ""),
  ("34", "MsgSeqNum", "1", ""),
  ("50", "SenderSubID", "30737", ""),
  ("97", "PossResend", "Y", "YES"),
  ("52", "SendingTime", "20000809-20:20:50", ""),
  ("11", "ClOrdID", "90001008", ""),
  ("1", "Account", "10030003", ""),
  ("21", "HandlInst", "2", "AUTOMATED_EXECUTION_INTERVENTION_OK"),
  ("55", "Symbol", "TESTA", ""),
  ("54", "Side", "1", "BUY"),
  ("38", "OrderQty", "4000", ""),
  ("40", "OrdType", "2", "LIMIT"),
  ("59", "TimeInForce", "0", "DAY"),
  ("44", "Price", "30", ""),
  ("47", "Rule80A", "I", "INDIVIDUAL_INVESTOR"),
  ("60", "TransactTime", "20000809-18:20:32", ""),
  ("10", "CheckSum", "061", "")
)

Show as dataframe

def showAsDF(msg:String, sep:Char='|'): Unit = {
    val p = parse(msg, sep)
    val cn = Seq("tag", "field name", "value", "description")

    println(f"${cn(0)}%6s${cn(1)}%30s${cn(2)}%40s${cn(3)}%50s")
    for (el <- p) println(f"${el(0)}%6s${el(1)}%30s${el(2)}%40s${el(3)}%50s")
}

Show as dataframe


fix: String = "8=FIX.4.2|9=153|35=D|49=BLP|56=SCHB|34=1|50=30737|97=Y|52=20000809-20:20:50|11=90001008|1=10030003|21=2|55=TESTA|54=1|38=4000|40=2|59=0|44=30|47=I|60=20000809-18:20:32|10=061|"


showAsDF(fix)

tag                    field name                                   value                                       description
  8                   BeginString                                 FIX.4.2                                                  
  9                    BodyLength                                     153                                                  
 35                       MsgType                                       D                                  NEW_ORDER_SINGLE
 49                  SenderCompID                                     BLP                                                  
 56                  TargetCompID                                    SCHB                                                  
 34                     MsgSeqNum                                       1                                                  
 50                   SenderSubID                                   30737                                                  
 97                    PossResend                                       Y                                               YES
 52                   SendingTime                       20000809-20:20:50                                                  
 11                       ClOrdID                                90001008                                                  
  1                       Account                                10030003                                                  
 21                     HandlInst                                       2               AUTOMATED_EXECUTION_INTERVENTION_OK
 55                        Symbol                                   TESTA                                                  
 54                          Side                                       1                                               BUY
 38                      OrderQty                                    4000                                                  
 40                       OrdType                                       2                                             LIMIT
 59                   TimeInForce                                       0                                               DAY
 44                         Price                                      30                                                  
 47                       Rule80A                                       I                               INDIVIDUAL_INVESTOR
 60                  TransactTime                       20000809-18:20:32                                                  
 10                      CheckSum                                     061

To ordered map

import scala.collection.mutable.LinkedHashMap

def toLinkedHashMap(msg:String, sep:Char='|') = {
    val o = LinkedHashMap[String,String]()
    parse(msg).map(
        x => o.addOne(x(1), if x(3).length > 0 then x(3) else x(2))
    )

    o
}

To ordered map


fix: String = "8=FIX.4.2|9=153|35=D|49=BLP|56=SCHB|34=1|50=30737|97=Y|52=20000809-20:20:50|11=90001008|1=10030003|21=2|55=TESTA|54=1|38=4000|40=2|59=0|44=30|47=I|60=20000809-18:20:32|10=061|"


toLinkedHashMap(fix)

res89: LinkedHashMap[String, String] = LinkedHashMap(
  "BeginString" -> "FIX.4.2",
  "BodyLength" -> "153",
  "MsgType" -> "NEW_ORDER_SINGLE",
  "SenderCompID" -> "BLP",
  "TargetCompID" -> "SCHB",
  "MsgSeqNum" -> "1",
  "SenderSubID" -> "30737",
  "PossResend" -> "YES",
  "SendingTime" -> "20000809-20:20:50",
  "ClOrdID" -> "90001008",
  "Account" -> "10030003",
  "HandlInst" -> "AUTOMATED_EXECUTION_INTERVENTION_OK",
  "Symbol" -> "TESTA",
  "Side" -> "BUY",
  "OrderQty" -> "4000",
  "OrdType" -> "LIMIT",
  "TimeInForce" -> "DAY",
  "Price" -> "30",
  "Rule80A" -> "INDIVIDUAL_INVESTOR",
  "TransactTime" -> "20000809-18:20:32",
  "CheckSum" -> "061"
)

When to engage the type system

The type system

Recap: we're still at String


fix: String = "8=FIX.4.2|9=153|35=D|49=BLP|56=SCHB|34=1|50=30737|97=Y|52=20000809-20:20:50|11=90001008|1=10030003|21=2|55=TESTA|54=1|38=4000|40=2|59=0|44=30|47=I|60=20000809-18:20:32|10=061|"


(fix.split('|') zip toLinkedHashMap(fix)).foreach { (l, r) => println(f"${l}%20s | \"${r(0)}\" -> \"${r(1)}\"") } 

           8=FIX.4.2 | "BeginString" -> "FIX.4.2"
               9=153 | "BodyLength" -> "153"
                35=D | "MsgType" -> "NEW_ORDER_SINGLE"
              49=BLP | "SenderCompID" -> "BLP"
             56=SCHB | "TargetCompID" -> "SCHB"
                34=1 | "MsgSeqNum" -> "1"
            50=30737 | "SenderSubID" -> "30737"
                97=Y | "PossResend" -> "YES"
52=20000809-20:20:50 | "SendingTime" -> "20000809-20:20:50"
         11=90001008 | "ClOrdID" -> "90001008"
          1=10030003 | "Account" -> "10030003"
                21=2 | "HandlInst" -> "AUTOMATED_EXECUTION_INTERVENTION_OK"
            55=TESTA | "Symbol" -> "TESTA"
                54=1 | "Side" -> "BUY"
             38=4000 | "OrderQty" -> "4000"
                40=2 | "OrdType" -> "LIMIT"
                59=0 | "TimeInForce" -> "DAY"
               44=30 | "Price" -> "30"
                47=I | "Rule80A" -> "INDIVIDUAL_INVESTOR"
60=20000809-18:20:32 | "TransactTime" -> "20000809-18:20:32"
              10=061 | "CheckSum" -> "061"

Cherry-pick a FIX message

           8=FIX.4.2 | "BeginString" -> "FIX.4.2"
               9=153 | "BodyLength" -> "153"
                35=D | "MsgType" -> "NEW_ORDER_SINGLE"
              49=BLP | "SenderCompID" -> "BLP"
             56=SCHB | "TargetCompID" -> "SCHB"
                34=1 | "MsgSeqNum" -> "1"
            50=30737 | "SenderSubID" -> "30737"
                97=Y | "PossResend" -> "YES"
52=20000809-20:20:50 | "SendingTime" -> "20000809-20:20:50"
         11=90001008 | "ClOrdID" -> "90001008"
          1=10030003 | "Account" -> "10030003"
                21=2 | "HandlInst" -> "AUTOMATED_EXECUTION_INTERVENTION_OK"
            55=TESTA | "Symbol" -> "TESTA"
                54=1 | "Side" -> "BUY"
             38=4000 | "OrderQty" -> "4000"
                40=2 | "OrdType" -> "LIMIT"
                59=0 | "TimeInForce" -> "DAY"
               44=30 | "Price" -> "30"
                47=I | "Rule80A" -> "INDIVIDUAL_INVESTOR"
60=20000809-18:20:32 | "TransactTime" -> "20000809-18:20:32"
              10=061 | "CheckSum" -> "061"

Cherry-pick a FIX message

           8=FIX.4.2 | "BeginString" -> "FIX.4.2"
                35=D | "MsgType" -> "NEW_ORDER_SINGLE"
              49=BLP | "SenderCompID" -> "BLP"
             56=SCHB | "TargetCompID" -> "SCHB"
                34=1 | "MsgSeqNum" -> "1"
52=20000809-20:20:50 | "SendingTime" -> "20000809-20:20:50"
            55=TESTA | "Symbol" -> "TESTA"
                54=1 | "Side" -> "BUY"
             38=4000 | "OrderQty" -> "4000"
               44=30 | "Price" -> "30"
60=20000809-18:20:32 | "TransactTime" -> "20000809-18:20:32"

Cherry-pick a FIX message

case class D(
    BeginString:String,               //            8=FIX.4.2 | "BeginString" -> "FIX.4.2"
    MsgType:String,                   //                 35=D | "MsgType" -> "NEW_ORDER_SINGLE"
    SenderCompID:String,              //               49=BLP | "SenderCompID" -> "BLP"
    TargetCompID:String,              //              56=SCHB | "TargetCompID" -> "SCHB"
    MsgSeqNum:Long,                   //                 34=1 | "MsgSeqNum" -> "1"
    SendingTime:String,               // 52=20000809-20:20:50 | "SendingTime" -> "20000809-20:20:50"
    Symbol:String,                    //             55=TESTA | "Symbol" -> "TESTA"
    Side:String,                      //                 54=1 | "Side" -> "BUY"
    OrderQty:Long,                    //              38=4000 | "OrderQty" -> "4000"
    Price:Option[Double],             //                44=30 | "Price" -> "30"              
    TransactTime:Option[String],      // 60=20000809-18:20:32 | "TransactTime" -> "20000809-18:20:32"
    MISSINGFIELD:Option[String]       //             <<< NEVER PRESENT >>>
)

Cherry-pick a FIX message

case class D(
    BeginString:String,               //            8=FIX.4.2 | "BeginString" -> "FIX.4.2"
    MsgType:String,                   //                 35=D | "MsgType" -> "NEW_ORDER_SINGLE"
    SenderCompID:String,              //               49=BLP | "SenderCompID" -> "BLP"
    TargetCompID:String,              //              56=SCHB | "TargetCompID" -> "SCHB"
    MsgSeqNum:Long,                   //                 34=1 | "MsgSeqNum" -> "1"
    SendingTime:String,               // 52=20000809-20:20:50 | "SendingTime" -> "20000809-20:20:50"
    Symbol:String,                    //             55=TESTA | "Symbol" -> "TESTA"
    Side:String,                      //                 54=1 | "Side" -> "BUY"
    OrderQty:Long,                    //              38=4000 | "OrderQty" -> "4000"
    Price:Option[Double],             //                44=30 | "Price" -> "30"              
    TransactTime:Option[String],      // 60=20000809-18:20:32 | "TransactTime" -> "20000809-18:20:32"
    MISSINGFIELD:Option[String]       //             <<< NEVER PRESENT >>>
)

object D {
    def apply(msg:LinkedHashMap[String,String]):D = {
        D(
            msg.getOrElse("BeginString", ""),          
            msg.getOrElse("MsgType", ""),              
            msg.getOrElse("SenderCompID", ""),
            msg.getOrElse("TargetCompID", ""),
            msg.getOrElse("MsgSeqNum", "").toLong,     // <- absurd hack! avoids map.get returning an Option[Long]
            msg.getOrElse("SendingTime", ""),
            msg.getOrElse("Symbol", ""),
            msg.getOrElse("Side", ""),   
            msg.getOrElse("OrderQty", "").toLong,      // <- absurd hack! avoids map.get returning an Option[Long] 
            msg.get("Price").map(_.toDouble),
            msg.get("TransactTime").map(_.toString),
            msg.get("MISSINGFIELD").map(_.toString),
        )
    }
}

Cherry-pick a FIX message


fix: String = "8=FIX.4.2|9=153|35=D|49=BLP|56=SCHB|34=1|50=30737|97=Y|52=20000809-20:20:50|11=90001008|1=10030003|21=2|55=TESTA|54=1|38=4000|40=2|59=0|44=30|47=I|60=20000809-18:20:32|10=061|"


D(toLinkedHashMap(fix))

res15: D = D(
  BeginString = "FIX.4.2",
  MsgType = "NEW_ORDER_SINGLE",
  SenderCompID = "BLP",
  TargetCompID = "SCHB",
  MsgSeqNum = 1L,
  SendingTime = "20000809-20:20:50",
  Symbol = "TESTA",
  Side = "BUY",
  OrderQty = 4000L,
  Price = Some(value = 30.0),
  TransactTime = Some(value = "20000809-18:20:32"),
  MISSINGFIELD = None
)

Is there a better way?

Cherry-pick a FIX message

case class D1(
    BeginString:String,               //            8=FIX.4.2 | "BeginString" -> "FIX.4.2"
    MsgType:String,                   //                 35=D | "MsgType" -> "NEW_ORDER_SINGLE"
    SenderCompID:String,              //               49=BLP | "SenderCompID" -> "BLP"
    TargetCompID:String,              //              56=SCHB | "TargetCompID" -> "SCHB"
    MsgSeqNum:Long,                   //                 34=1 | "MsgSeqNum" -> "1"
    SendingTime:String,               // 52=20000809-20:20:50 | "SendingTime" -> "20000809-20:20:50"
    Symbol:String,                    //             55=TESTA | "Symbol" -> "TESTA"
    Side:String,                      //                 54=1 | "Side" -> "BUY"
    OrderQty:Long,                    //              38=4000 | "OrderQty" -> "4000"
    Price:Option[Double],             //                44=30 | "Price" -> "30"              
    TransactTime:Option[String],      // 60=20000809-18:20:32 | "TransactTime" -> "20000809-18:20:32"
    MISSINGFIELD:Option[String]       //             <<< NEVER PRESENT >>>
)

object D1 {
    val classFields = classOf[D1].getDeclaredFields.map(_.getName).toList

    def apply(msg:LinkedHashMap[String,String]):D1 = {
        val seq = classFields.map(field => msg.getOrElse(field, ""))

        Decoder.toProduct[D1](seq)
    }

}

Cherry-pick a FIX message


fix: String = "8=FIX.4.2|9=153|35=D|49=BLP|56=SCHB|34=1|50=30737|97=Y|52=20000809-20:20:50|11=90001008|1=10030003|21=2|55=TESTA|54=1|38=4000|40=2|59=0|44=30|47=I|60=20000809-18:20:32|10=061|"


D1(toLinkedHashMap(fix))

res1: D1 = D1(
  BeginString = "FIX.4.2",
  MsgType = "NEW_ORDER_SINGLE",
  SenderCompID = "BLP",
  TargetCompID = "SCHB",
  MsgSeqNum = 1L,
  SendingTime = "20000809-20:20:50",
  Symbol = "TESTA",
  Side = "BUY",
  OrderQty = 4000L,
  Price = Some(value = 30.0),
  TransactTime = Some(value = "20000809-18:20:32"),
  MISSINGFIELD = None
)

Type classes to the rescue!

// Source: based on https://github.com/yummydum/scala3Practice/blob/main/src/main/scala/myutils/utils.scala
import scala.deriving.Mirror

object Decoder {
    type Row = Seq[String]

    trait FieldDecoder[A]:
        def decodeField(a: String): A

    trait RowDecoder[A <: Tuple]:
        def decodeRow(a: Row): A

    given FieldDecoder[Int] with
        def decodeField(x: String) = x.toInt

    given FieldDecoder[Boolean] with
        def decodeField(x: String) = x.toBoolean

    given FieldDecoder[Long] with
        def decodeField(x: String) = x.toLong

    given FieldDecoder[Double] with
        def decodeField(x: String) = x.toDouble

    given fdod:FieldDecoder[Option[Double]] with
        def decodeField(x: String) = if x.length > 0 then Some(x.toDouble) else None

    given FieldDecoder[String] with
        def decodeField(x: String) = x

    given fdos:FieldDecoder[Option[String]] with
        def decodeField(x: String) = if x.length > 0 then Some(x) else None

    given RowDecoder[EmptyTuple] with
        def decodeRow(row: Row) = EmptyTuple

    // Type parameter :
    //     H: FieldDecoder, T <: Tuple: RowDecoder
    // Given instance for :
    //     RowDecoder[H *: T]
    given [H: FieldDecoder, T <: Tuple: RowDecoder]: RowDecoder[H *: T] with
        def decodeRow(row: Row) = {
            summon[FieldDecoder[H]].decodeField(row.head) *: summon[RowDecoder[T]]
            .decodeRow(row.tail)
        }

    def toProduct[P](row: Row)(using p: Mirror.ProductOf[P], d: RowDecoder[p.MirroredElemTypes]): P =
        p.fromProduct(d.decodeRow(row))
}

Parsimony of Scala

  • the examples in this talk amount to a mere

    • 229 lines of code 

      wc -l functions.sc

      229 functions.sc
    • the core functional code is only about 1/2 of that
  • the basis of six - a Scala package I'm writing