Mr. Marius Gligor

I'm an experienced Software Developer and I provide Software Development and Consulting Services in the following areas of expertise:

Scala, Java, Python, C, C++ Software Development services.
Design and implements Scala microservices with Akka, Akka Actors, Akka-HTTP, Akka Streams.
Design and implements Scala microservices with Cats, Monix, http4s, circe, fs2.
Design and implements Scala microservices with Play framework.
Design and implements JavaScript (ReactJS), HTML, CSS front-end applications.
Implements Scala persistence layers for RDBMS databases with Slick, Doobie.
Implements Scala persistence layers for Cassandra, MongoDB NoSQL databases.
Design and implements Java microservices with Spring Boot and Spring Cloud.
Implements Java persistence layers for RDBMS databases with JDBC, Hibernate, JPA and Spring Data.
Design and implements Python microservices with Django, Falcon.
Other IT Consulting Services.

Why I love Scala!

Scala is a general purpose programming language designed to express common programming patterns in a concise, elegant, and type-safe way. It smoothly integrates features of object-oriented and functional languages, enabling Java and other programmers to be more productive. Code sizes are typically reduced by a factor of two to three when compared to an equivalent Java application.

// safe factorial
def factorial(n: BigInt): BigInt =
  (BigInt(1) to n).foldLeft(BigInt(1))((f, v) => f * v)

def factorial(n: BigInt): BigInt =
  (BigInt(1) to n).product

// quick sort
def sort(data: List[Int]): List[Int] = data match {
  case Nil | _ :: Nil => data
  case x :: xs => sort(xs.filter(_ < x)) ++ List(x) ++ sort(xs.filter(_ >= x))
}

// compute min and max values
def min(data: Array[Int]) = {
    assert(data.nonEmpty)
    data.foldLeft(data(0))((m, i) => if (i < m) i else m)
}

def max(data: Array[Int]) = {
    assert(data.nonEmpty)
    data.foldLeft(data(0))((m, i) => if (i > m) i else m)
}

Type Safe Equality in Scala

Below is a simple solution of type safe equality implementation in Scala.

trait Equal[A] {
  def ===(a: A): Boolean
  def =/=(a: A): Boolean
}

implicit case class Equals[A](a: A) extends Equal[A] {
  override def ===(b: A): Boolean = a == b
  override def =/=(b: A): Boolean = a != b
}

println("123" == 123) // print false
println(123 == "123") // print false
println("123" === 123) // compile error (type mismatch)
println(123 === "123") // compile error (type mismatch)

println("123" === "123") // print true
println("123" =/= "123") // print false
println("123" === "124") // print false
println("123" =/= "124") // print true

Linear Regression

Linear regression is a basic and commonly used type of predictive analysis. Below is a simple Scala implementation.

class LinearRegression (a: Double, b: Double) {
  def forecast(p: Double): Double = a + b * p
}

object LinearRegression {
  private def avg(data: Array[(Double, Double)]) = {
    val length = data.length
    assert(length > 0)
    val (sum1, sum2) = data.foldLeft((0.0, 0.0)) { (t, v) =>
      (t._1 + v._1, t._2 + v._2)
    }
    (sum1 / length, sum2 / length)
  }

  private def coeffs(data: Array[(Double, Double)]) = {
    val (xm, ym) = avg(data)
    val (sum1, sum2) = data.foldLeft((0.0, 0.0)) { (t, v) =>
      val xv = v._1 - xm
      val yv = v._2 - ym
      (t._1 + xv * yv, t._2 + xv * xv)
    }
    val b = sum1 / sum2
    val a = ym - xm * b
    (a, b)
  }

  def apply(data: Array[(Double, Double)]): LinearRegression = {
    val (a, b) = coeffs(data)
    new LinearRegression(a, b)
  }
}

object Predict extends App {
  val x = Array[Double](1, 2, 3, 4, 5)
  val y = Array[Double](4.8730, 4.8800, 4.8750, 4.8678, 4.8690)
  val forecaster = LinearRegression(x zip y)
  println(forecaster.forecast(6)) // yield 4.8669
}

Logistic Regression

In statistics, the logistic model (or logit model) is used to model the probability of a certain class or event existing such as pass/fail, win/lose, alive/dead or healthy/sick.

case class LogisticRegression(L: Double = 1.0, a: Double = 1.0, b: Double = 0.0) {

  def logistic(x: Double): Double = {
    val p = L / (1.0 + Math.exp(-a * (x - b)))
    if (p < 0.5) 0.0 else 1.0
  }

  def logit(p: Double): Double =
    -Math.log((1.0 / p) - 1.0)
}

Normal Distribution

Normal distribution, also known as the Gaussian distribution, is a probability distribution that is symmetric about the mean, showing that data near the mean are more frequent in occurrence than data far from the mean. In graph form, normal distribution will appear as a bell curve.

object ZScore {

  def zScore(x: Double, mean: Double, sigma: Double, size: Int = 1): Double = {
    val score = (x - mean) / sigma
    if (size == 1) score else score * Math.sqrt(size)
  }
}

object PValue {

  def pLeftValue(x: Double, mean: Double, sigma: Double, size: Int = 1): Double =
    CDF(x, mean, sigma, size)

  def pRightValue(x: Double, mean: Double, sigma: Double, size: Int = 1): Double =
    1.0 - CDF(x, mean, sigma, size)

  def pTwoSidedValue(x: Double, mean: Double, sigma: Double, size: Int = 1): Double =
    2.0 * Math.min(CDF(x, mean, sigma, size), 1.0 - CDF(x, mean, sigma, size))
}

object NormalDistribution {

  def erf(x: Double): Double = {
    val a1 = 0.254829592
    val a2 = -0.284496736
    val a3 = 1.421413741
    val a4 = -1.453152027
    val a5 = 1.061405429
    val p = 0.3275911
    val t = 1.0 / (1.0 + p * Math.abs(x))
    1.0 - ((((((a5 * t + a4) * t) + a3) * t + a2) * t) + a1) * t * Math.exp(-1.0 * x * x)
  }

  def CDF(x: Double, mean: Double, sigma: Double, size: Int = 1): Double = {
    val z = zScore(x, mean, sigma, size)
    val sign = if (z < 0) -1.0 else 1.0
    0.5 * (1.0 + sign * erf(0.70710678118 * Math.abs(z)))
  }

  def PDF(x: Double, mean: Double, sigma: Double): Double = {
    val z = zScore(x, mean, sigma)
    0.3989422804 * Math.exp(-0.5 * z * z) / sigma
  }
}

Interpolation

Interpolation is a type of estimation, a method of constructing new data points within the range of a discrete set of known data points.

Lagrange polynomials interpolation

case class Lagrange(data: Array[(Double, Double)]) {

  private val n = data.length

  def interpolate(xi: Double): Double = {
    (0 until n).foldLeft(0.0) { (result, i) =>
      result + (0 until n).filter(_ != i).foldLeft(data(i)._2) { (term, j) =>
        term * (xi - data(j)._1) / (data(i)._1 - data(j)._1)
      }
    }
  }

  def inverse(yi: Double): Double = {
    (0 until n).foldLeft(0.0) { (result, i) =>
      result + (0 until n).filter(_ != i).foldLeft(data(i)._1) { (term, j) =>
        term * (yi - data(j)._2) / (data(i)._2 - data(j)._2)
      }
    }
  }
}

Newton's divided difference interpolation

case class Newton(data: Array[(Double, Double)]) {

  private val n = data.length

  // initialize
  val ddt: Array[Array[Double]] = Array.ofDim[Double](n, n)
  (0 until n).foreach(i => ddt(i)(0) = data(i)._2)

  // build divided diff table
  (1 until n) foreach { i =>
    (0 until n - i) foreach { j =>
      ddt(j)(i) = (ddt(j)(i - 1) - ddt(j + 1)(i - 1)) / (data(j)._1 - data(i + j)._1)
    }
  }

  def interpolate(xi: Double): Double = {
    // Function to find the product term
    def prodterm(i: Int): Double = {
      (0 until i).foldLeft(1.0) { (pro, j) =>
        pro * (xi - data(j)._1)
      }
    }

    (1 until n).foldLeft(ddt(0)(0)) { (sum, i) =>
      sum + prodterm(i) * ddt(0)(i)
    }
  }
}

Bessel central difference interpolation

case class Bessel(data: Array[(Double, Double)]) {

  private val n = data.length

  // initialize
  val cdt: Array[Array[Double]] = Array.ofDim[Double](n, n)
  (0 until n).foreach(i => cdt(i)(0) = data(i)._2)

  // build central difference table
  (1 until n) foreach { i =>
    (0 until n - i) foreach { j =>
      cdt(j)(i) = cdt(j + 1)(i - 1) - cdt(j)(i - 1)
    }
  }

  def interpolate(xi: Double): Double = {
    def fact(n: Int) = (1 to n).product

    def isEven(number: Int) = number % 2 == 0

    def ucal(u: Double, n: Int): Double = {
      if (n == 0) {
        1.0
      } else {
        val v1 = (1 to n / 2).foldLeft(u)((temp, i) => temp * (u - i))
        (1 until n / 2).foldLeft(v1)((temp, i) => temp * (u + i))
      }
    }

    // Initializing u and sum
    val k = new AtomicInteger(if (isEven(n)) n / 2 - 1 else n / 2)
    val sum = (cdt(k.get())(0) + cdt(k.get() + 1)(0)) / 2
    val u = (xi - data(k.get())._1) / (data(1)._1 - data(0)._1)

    // Solving using bessel's formula
    (1 until n).foldLeft(sum) { (s, i) =>
      s + {
        if (isEven(i))
          ucal(u, i) * (cdt(k.get())(i) + cdt(k.decrementAndGet())(i)) / (fact(i) * 2)
        else
          (u - 0.5) * ucal(u, i - 1) * cdt(k.get())(i) / fact(i)
      }
    }
  }
}

Functional Programming (FP) type classes in Scala

Unlike Java, Scala is a programming language pure Object-Oriented (OO) which combine OO code with FP style code. Depending on your choice you can write pure OO code, mixed OO with FP code or pure FP code. Nevertheless, the power of Scala is remarkable when you write FP style code. When you write FP code you have to work with type classes like Monoids, Functors, Monads, etc., which are coming from Category Theory in mathematics. These data structures are not "first citizen" classes in Scala. However, some Scala classes like Option, Future, List are designed as monads. Monadic composition in Scala is implemented using the for comprehension syntactic sugar. If you are an advanced Scala developer writing pure functional code, you must use libraries like Cats, Monix, ZIO, Scalaz, http4s, fs2, circe, argonaut, shapeless, monocle, doobie etc. Also, you can try to learn about Haskell, an advanced, purely functional programming language, which is the reference in functional programming.

trait Semigroup[A] {
  def combine(x: A, y: A): A
}

trait Monoid[A] extends Semigroup[A] {
  def empty: A
}

trait Functor[F[_]] {
  def map[A, B](fa: F[A])(f: A => B): F[B]
}

trait Apply[F[_]] {
  def ap[A, B](fa: F[A])(ff: F[A => B]): F[B]
}

trait Applicative[F[_]] extends Functor[F] with Apply[F] {
  def pure[A](a: A): F[A]
  def map[A, B](fa: F[A])(f: A => B): F[B] = ap(fa)(pure(f))
}

trait FlatMap[F[_]] {
  def flatMap[A, B](fa: F[A])(f: A => F[B]): F[B]
}

trait Monad[F[_]] extends Applicative[F] with FlatMap[F] {
  /** map can be implemented using flatMap and pure */
  override def map[A, B](fa: F[A])(f: A => B): F[B] = flatMap(fa)(a => pure(f(a)))
}

trait Choice[F[_, _]] {
  def choice[A, B, C](fac: F[A, C], fbc: F[B, C]): F[Either[A, B], C]
}

trait ArrowChoice[F[_, _]] extends Choice[F] {
  def choose[A, B, C, D](f: F[A, C])(g: F[B, D]): F[Either[A, B], Either[C, D]]
}

trait Bifunctor[F[_, _]] {
  def bimap[A, B, C, D](fab: F[A, B])(f: A => C, g: B => D): F[C, D]
}

trait Invariant[F[_]] {
  def imap[A, B](fa: F[A])(f: A => B)(g: B => A): F[B]
}

trait Contravariant[F[_]] {
  def contramap[A, B](fa: F[A])(f: B => A): F[B]
}

final case class Reader[A, B](run: A => B)
// type Reader[A, B] = Kleisli[Id, A, B]
final case class Kleisli[F[_], A, B](run: A => F[B]) // ReaderT

final case class State[S, +A](run: S => (S, A))
// type State[S, +A] = StateT[Id, S, A]
final case class StateT[F[_], S, +A](runF: F[S => F[(S, A)]])

Semigroup

In functional programming, a Semigroup is an algebraic data structure which define an associative binary operation for a type A: combine(x, combine(y, z)) = combine(combine(x, y), z). Below is a small example which shows how we can use a Semigroup in our Scala code.

case class Point(x: Double, y: Double)

implicit val pointSemigroup: Semigroup[Point] =
  Semigroup.instance[Point]((p1, p2) => Point(p1.x + p2.x, p1.y + p2.y))

assert((Point(1.0, 2.0) |+| Point(3.0, 4.0) |+| Point(-1.0, -1.0)) == Point(3.0, 5.0))
assert((Point(1.0, 2.0).some |+| Point(3.0, 4.0).some |+| Point(-1.0, -1.0).some) == Point(3.0, 5.0).some)
assert((Point(1.0, 2.0).some |+| none |+| Point(-1.0, -1.0).some) == Point(0.0, 1.0).some)

Making our own typeclass with simulacrum

Making our own typeclass is easy to do with simulacrum annotations @typeclass, @op Bellow is an example for encoding / decoding bytes.

@typeclass
trait ByteEncoder[A] {
  def encode(a: A): Either[Throwable, Array[Byte]]
}

object ByteEncoder {
  implicit val stringEncoder: ByteEncoder[String] =
    a => Try(a.getBytes("UTF-8")).toEither
}

@typeclass
trait ByteDecoder[A] {
  def decode(a: A): Either[Throwable, String]
}

object ByteDecoder {
  implicit val bytesDecoder: ByteDecoder[Array[Byte]] =
    a => Try(new String(a, "UTF-8")).toEither
}

object ByteCodec extends App {
  import ByteDecoder.ops._
  import ByteEncoder.ops._

  val bytes = "The quick brown fox jumps over the lazy dog".encode
  println(bytes)

  bytes match {
    case Left(value)  => value.printStackTrace()
    case Right(value) => println(value.decode)
  }
}

The same functionality can be achieved in Scala 3 using implicit conversions.

object Converter:
  given Conversion[String, Either[Throwable, Array[Byte]]] with
    def apply(a: String) = Try(a.getBytes("UTF-8")).toEither

  given Conversion[Array[Byte], Either[Throwable, String]] with
    def apply(a: Array[Byte]) = Try(new String(a, "UTF-8")).toEither

object Main extends App:
  import Converter.given

  val bytes = "The quick brown fox jumps over the lazy dog".convert
  println(bytes)

  bytes match
    case Left(th)  => th.printStackTrace()
    case Right(value) => println(value.convert)

ore more concisely as:

object Converter:
  given Conversion[String, Either[Throwable, Array[Byte]]] = a => Try(a.getBytes("UTF-8")).toEither

  given Conversion[Array[Byte], Either[Throwable, String]] = a => Try(new String(a, "UTF-8")).toEither

Value classes and Scala 3 opaque types aliases

Value classes are a mechanism in Scala to avoid allocating runtime objects. This is accomplished through the definition of new AnyVal subclasses. The following shows a value class example:

case class Year(value: Int) extends AnyVal {
  def safe: Option[Year] = if (value > 2000) Some(this) else None
}

@main def main: Unit =

  val year: Year = Year(2000)
  assert(year.value == 2000)

  val spaceOdyssey = for {
    year <- Year(1968).safe
  } yield year
  assert (spaceOdyssey.isEmpty)

However, because the JVM does not support value classes, Scala sometimes needs to actually instantiate a value class. Scala 3 opaque types aliases provide type abstraction without any overhead.

object opaques:
  opaque type Year = Int

  object Year:
    // set value
    def apply(value: Int): Year = value

    // safe operations (validation)
    def safe(value: Int): Option[Year] = if (value > 2000) Some(value) else None

  // access value
  extension (year: Year)
    def value: Int = year

@main def main: Unit =
  import opaques._

  val year: Year = Year(2000)
  assert(year.value == 2000)

  val spaceOdyssey = for {
    year <- Year.safe(1968)
  } yield year
  assert (spaceOdyssey.isEmpty)

Monad transformers

Transforms nested Monads into one monad. Below are some potential implementations for Option and Either

case class OptionT[F[_], A](value: F[Option[A]])(implicit F: Monad[F]) {

  def map[B](f: A => B): OptionT[F, B] =
    OptionT(F.map(value)(a => a.map(f)))

  def flatMap[B](f: A => OptionT[F, B]): OptionT[F, B] =
    OptionT(F.flatMap(value) {
      case Some(a) => f(a).value
      case None    => F.pure(None)
    })
}

case class EitherT[F[_], A](value: F[Either[Throwable, A]])(implicit F: Monad[F]) {

  def map[B](f: A => B): EitherT[F, B] =
    EitherT(F.map(value)(a => a.map(f)))

  def flatMap[B](f: A => EitherT[F, B]): EitherT[F, B] =
    EitherT(F.flatMap(value)(_.fold(l => F.pure(l), r => f(r).value)))
}

A generic monad transformer is not possible to be implemented because Monads don't compose, at least not generically.

case class MonadT[F[_], M[_], A](value: F[M[A]])(implicit F: Monad[F], M: Monad[M]) {

  def map[B](f: A => B): MonadT[F, M, B] =
    MonadT(F.map(value)(ma => M.map(ma)(f)))

  def flatMap[B](f: A => MonadT[F, M, B]): MonadT[F, M, B] = ???
}

Handle errors with MonadError

Error handling has been always a problem not entirely solved in software development. MonadError is a type class that abstracts over error-handling monads, allowing to raise and/or handle an error value.

def validate[F[_], A](f: => A)(implicit M: MonadError[F, Throwable]): F[A] =
  Try(f).toEither match {
    case Right(a) => M.pure(a)
    case Left(error) => M.raiseError(error)
  }

def program(a: => Int): IO[Int] = (for {
  result <- validate[IO, Int](a)
} yield result)
  .handleErrorWith { error =>
    println(s"An error occurred while evaluating expression, $error")
    IO.raiseError(error)
  }

// will print "An error occurred while evaluating expression, java.lang.ArithmeticException: / by zero"
program(5 / 0).unsafeRunSync()

Tagless final design pattern in Scala

The best definition of Tagless final by John A De Goes:

"Tagless-final is a technique originally used to embed domain-specific languages into a host language, without the use of Generalized Algebraic Data Types."

The first step in tagless final pattern implementation is to define an algebra, for our new language, as abstract methods in a trait parameterized with a type constructor. The results of algebra methods are wrapped inside the type constructor. Let us use the well known Console example which is the tagless final equivalent of Hello World! example.

trait Console[F[_]] {
  def readLn: F[String]
  def putStrLn(text: String): F[Unit]
}

Next, based on the defined algebra we will write a declarative program. The program dependencies as implicit values could be provided in two ways, as parameters or as context bound.

Dependencies as implicit parameters:

def program[F[_]: Monad](implicit C: Console[F]): F[Unit] =
  for {
    _ <- C.putStrLn("Enter your name:")
    n <- C.readLn
    _ <- C.putStrLn(s"Hello $n!")
  } yield ()

Dependencies as context bound:

object Console {
  def apply[F[_]](implicit console: Console[F]): Console[F] = console
}

def program[F[_]: Console: Monad]: F[Unit] =
  for {
    _ <- Console[F].putStrLn("Enter your name:")
    n <- Console[F].readLn
    _ <- Console[F].putStrLn(s"Hello $n!")
  } yield ()

Both programs are equivalent and describe a computation. Both are abstract and does nothing. However, programs can be composed with other abstract programs, resulting new abstract programs. In order to execute effects, we must provide interpreters, that is, concrete implementation for all defined algebras, in our case for Console algebra. A possible implementation of a standard console using Cats IO monad looks like:

object ConsoleInterpreter {
  val standardConsole: Console[IO] = new Console[IO] {
    def putStrLn(text: String): IO[Unit] = IO(println(text))
    def readLn: IO[String] = IO(scala.io.StdIn.readLine)
  }
}

Finally, we have to create a concrete program, provide an instance of a console interpreter, in our case the standard console, and execute the program.

implicit val standardConsole: Console[IO] = ConsoleInterpreter.standardConsole

val concreteProgram: IO[Unit] =
  for {
    _ <- program[IO]
  } yield ()

concreteProgram.unsafeRunSync()

Scala 3 implicits

In Scala 3 (Dotty) version, implicits are replaced by the given / using declarations.

object Console:
  def apply[F[_]](using console: Console[F]): Console[F] = console

def program[F[_] : Console : Monad]: F[Unit] = for {
  _ <- Console[F].putStrLn("Enter your name:")
  n <- Console[F].readLn
  _ <- Console[F].putStrLn(s"Hello $n!")
} yield ()

def program[F[_] : Monad](using C: Console[F]): F[Unit] = for {
  _ <- C.putStrLn("Enter your name:")
  n <- C.readLn
  _ <- C.putStrLn(s"Hello $n!")
} yield ()

given standardConsole: Console[IO] with
  def putStrLn(text: String): IO[Unit] = IO(println(text))
  def readLn: IO[String] = IO(scala.io.StdIn.readLine)

Scala 3 enforces some rules on indentation and allows some occurrences of braces to be optional.

object TagLess extends App :

  trait Console[F[_]]:
    def readLn: F[String]
    def putStrLn(text: String): F[Unit]
  end Console

  object Console:
    def apply[F[_]](using console: Console[F]): Console[F] = console
  end Console

  given standardConsole: Console[IO] with
    def putStrLn(text: String): IO[Unit] = IO(println(text))
    def readLn: IO[String] = IO(scala.io.StdIn.readLine)

  def program[F[_] : Monad](using C: Console[F]): F[Unit] =
    for
      _ <- C.putStrLn("Enter your name:")
      n <- C.readLn
      _ <- C.putStrLn(s"Hello $n!")
    yield ()

  def programV2[F[_] : Console : Monad]: F[Unit] =
    for
      _ <- Console[F].putStrLn("Enter your name:")
      n <- Console[F].readLn
      _ <- Console[F].putStrLn(s"Hello $n!")
    yield ()

  val concreteProgram: IO[Unit] =
    for
      _ <- program[IO]
    yield ()

  concreteProgram.unsafeRunSync()
end TagLess

Cats-tagless library

In conclusion tagless final is very easy to be implemented. Additional we can use cats-tagless which is a library of utilities for tagless final algebras. For example, cats-tagless provides a FunctorK type class to map over algebras.

trait FunctorK[Alg[_[_]]] extends InvariantK[Alg] {
  def mapK[F[_], G[_]](af: Alg[F])(fk: F ~> G): Alg[G]
  override def imapK[F[_], G[_]](af: Alg[F])(fk: F ~> G)(gk: G ~> F): Alg[G] = mapK(af)(fk)
}

Below is an example of using FunctorK.mapK to map an algebra from Try to Option:

trait ExpressionAlg[F[_]] {
  def num(i: String): F[Float]
  def divide(dividend: Float, divisor: Float): F[Float]
}

implicit object tryExpression extends ExpressionAlg[Try] {
  def num(i: String): Try[Float] = Try(i.toFloat)
  def divide(dividend: Float, divisor: Float): Try[Float] = Try(dividend / divisor)
}

implicit val functorK: FunctorK[ExpressionAlg] =
  Derive.functorK[ExpressionAlg]

val fk: Try ~> Option = new (Try ~> Option) {
  def apply[A](t: Try[A]): Option[A] = t.toOption
}

val optionExpression: ExpressionAlg[Option] = tryExpression.mapK(fk)

Other functions example to map between Cats IO, Monix Task and Scala Future:

val fk: Future ~> IO = new (Future ~> IO) {
  def apply[A](t: Future[A]): IO[A] = IO.fromFuture(IO.delay(t))
}

val fk: IO ~> Future = new (IO ~> Future) {
  def apply[A](t: IO[A]): Future[A] = t.unsafeToFuture()
}

val fk: Future ~> Task = new (Future ~> Task) {
  def apply[A](t: Future[A]): Task[A] = Task.deferFuture(t)
}

val fk: Task ~> Future = new (Task ~> Future) {
  def apply[A](t: Task[A]): Future[A] = t.runToFuture
}

FunctorK.imapK example:

type EitherA[A] = Either[Throwable, A]

val fke: Try ~> EitherA = new (Try ~> EitherA) {
  def apply[A](t: Try[A]): EitherA[A] = t.toEither
}

val fkeContra: EitherA ~> Try = new (EitherA ~> Try) {
  def apply[A](e: EitherA[A]): Try[A] =
    e.fold(th => Failure(th), a => Success(a))
}

val invariantEitherExpression: ExpressionAlg[EitherA] = tryExpression.imapK(fke)(fkeContra)
val eitherExpression: ExpressionAlg[EitherA] = tryExpression.mapK(fke)

Horizontal composition. You can use the SemigroupalK type class to create a new interpreter that runs two interpreters simultaneously and return the result as a cats.Tuple2K

implicit val semigroupalK: SemigroupalK[ExpressionAlg] =
  Derive.semigroupalK[ExpressionAlg]

val product: ExpressionAlg[({type λ[*] = Tuple2K[Option, Try, *]})#λ] =
  optionExpression.productK(tryExpression)

Recently, John A De Goes, the author of the ZIO library, claims "The Death of Final Tagless". An interesting article regarding this topic was posted by Adam Warski here. ZIO introduces another pattern using modules and layers based on union of types.

Free Monads

A free monad is a construction which allows you to build a monad from any Functor. Free monad has the same expressive power like Tagless final. Again we have to define a grammar or DSL (Domain Specific Language) but this time using ADT's (Algebraic Data Type) instead abstract methods. Based on defined grammar we can build abstract programs. Next we have to write a compiler for our program and finally run it. Below is the same Console implementation with free monads.

sealed trait ConsoleOp[+A]

case class PrintLn(text: String) extends ConsoleOp[Unit]
case class ReadLn() extends ConsoleOp[String]

object Console {

  type Console[A] = Free[ConsoleOp, A]

  def readLn: Console[String] = Free.liftF(ReadLn())

  def printLn(text: String): Console[Unit] = Free.liftF(PrintLn(text))
}

object StandardConsoleCompiler {

  import Console._

  def apply[A](f: Console[A]): Id[A] = f.foldMap(Compiler)

  object Compiler extends (ConsoleOp ~> Id) {
    override def apply[A](fa: ConsoleOp[A]): Id[A] = fa match {
        case PrintLn(text) ⇒ println(text)
        case ReadLn() ⇒ scala.io.StdIn.readLine()
      }
  }
}

object ConsoleFreeMonad {

  import Console._

  lazy val program: Console[String] =
    for {
      _ <- printLn("Please tell me your name (empty to exit):")
      greeting = "Hello"
      name <- readLn
      _ <- printLn(s"$greeting $name")
    } yield name

  def main(args: Array[String]): Unit = {
    StandardConsoleCompiler(program)
  }
}

AWS S3 interface

Below is an example of Tagless final implementation for AWS S3 interface. First we define an S3 algebra:

sealed trait S3Algebra[F[_]] {

  def s3client(accessKeyId: String, secretAccessKey: String): F[AmazonS3]

  def createBucket(client: AmazonS3, bucketName: String): F[Bucket]

  def upload(client: AmazonS3, bucketName: String, key: String, file: File, isPublic: Boolean): F[PutObjectResult]

  def download(client: AmazonS3, bucketName: String, key: String): F[S3Object]

  def getResourceUrl(client: AmazonS3, bucketName: String, key: String): F[URL]

  def getObjectSummary(client: AmazonS3, bucketName: String, prefix: String): F[Option[S3ObjectSummary]]
}

Based on defined S3 algebra we create some abstract programs:

case class S3Programs[F[_] : Monad](accessKeyId: String, secretAccessKey: String)(implicit S3: S3Algebra[F]) {

  val s3client: F[AmazonS3] =
    for {
      client <- S3.s3client(accessKeyId, secretAccessKey)
    } yield client

  def download(bucketName: String, key: String): F[S3Object] =
    for {
      client <- s3client
      obj <- S3.download(client, bucketName, key)
    } yield obj

  def upload(bucketName: String, key: String, file: File, isPublic: Boolean): F[PutObjectResult] =
    for {
      client <- s3client
      summary <- S3.getObjectSummary(client, bucketName, key)
      _ = if (summary.isEmpty) S3.createBucket(client, bucketName)
      obj <- S3.upload(client, bucketName, key, file, isPublic)
    } yield obj
}

Next, we provide an interpreter which is a concrete implementation of our S3 algebra, as an implicit object.

implicit object S3CatsIOAlgebra extends S3Algebra[IO] {

  def s3client(accessKeyId: String, secretAccessKey: String): IO[AmazonS3] = IO {
    val credentials = new BasicAWSCredentials(accessKeyId, secretAccessKey)
    val provider = new AWSStaticCredentialsProvider(credentials)
    AmazonS3ClientBuilder.standard()
      .withCredentials(provider)
      .build()
  }

  def createBucket(client: AmazonS3, bucketName: String): IO[Bucket] = IO {
    client.createBucket(bucketName)
  }

  def upload(client: AmazonS3, bucketName: String, key: String, file: File, isPublic: Boolean): IO[PutObjectResult] = IO {
    val request = new PutObjectRequest(bucketName, key, file)
      .withCannedAcl(access(isPublic))
    client.putObject(request)
  }

  def download(client: AmazonS3, bucketName: String, key: String): IO[S3Object] = IO {
    client.getObject(new GetObjectRequest(bucketName, key))
  }

  def getResourceUrl(client: AmazonS3, bucketName: String, key: String): IO[URL] =
    IO(client.getUrl(bucketName, key))

  def getObjectSummary(client: AmazonS3, bucketName: String, prefix: String): IO[Option[S3ObjectSummary]] = IO {
    client.listObjects(bucketName, prefix).getObjectSummaries.asScala.toList.headOption
  }

  private def access(isPublic: Boolean): CannedAccessControlList =
    if (isPublic) CannedAccessControlList.PublicRead else CannedAccessControlList.Private
}

Finally let runs our programs.

val programs = S3Programs[IO]("accesskeyid", "accesskeysecret")
val obj = programs.download("bucketname", "key").unsafeRunSync()

Trampoline

Scala does tail recursion optimisation at compile-time, but for a self function only. FP is about functions composition which consume stack frames and a stack overflow could occur at runtime. One solution of this problem is to use trampoline optimisation. A simple trampoline implementation based on this paper Stackless Scala With Free Monads is described below. Also, Scala standard library provide the TailCalls object as a trampoline implementation.

sealed trait Trampoline[+A] {
  @tailrec
  final def runT: A =
    this match {
      case More(k) => k().runT
      case Done(v) => v
    }
}

case class More[+A](k: () => Trampoline[A]) extends Trampoline[A]
case class Done[+A](result: A) extends Trampoline[A]

The following snippet of code yield a stack overflow error when is executed.

def even[A](ns: List[A]): Id[Boolean] =
  ns match {
    case Nil     => true
    case x :: xs => odd(xs)
  }

def odd[A](ns: List[A]): Id[Boolean] =
  ns match {
    case Nil     => false
    case x :: xs => even(xs)
  }
}

val ns = (1 to 100000).toList
odd(ns) /* yield java.lang.StackOverflowError */
even(ns) /* yield java.lang.StackOverflowError */

The same code optimized with trampoline works fine!

def even[A](ns: List[A]): Trampoline[Boolean] =
  ns match {
    case Nil     => Done(true)
    case x :: xs => More(() => odd(xs))
  }

def odd[A](ns: List[A]): Trampoline[Boolean] =
  ns match {
    case Nil     => Done(false)
    case x :: xs => More(() => even(xs))
  }

val ns = (1 to 100000).toList
odd(ns).runT /* works fine! */
even(ns).runT /* works fine! */

How it works! Basically we reduce the alternate execution of methods even and odd to an execution of the trampoline runT method which is tail recursion optimized.

The same example using the Scala TailCalls object, and not a surprise, result method is tail recursion optimized:

import scala.util.control.TailCalls._

def isEven[A](ns: List[A]): TailRec[Boolean] =
  ns match {
    case Nil => done(true)
    case x::xs => tailcall(isOdd(xs))
  }

def isOdd[A](ns: List[A]): TailRec[Boolean] =
  ns match {
    case Nil => done(false)
    case x::xs => tailcall(isEven(xs))
  }

val ns = (1 to 100000).toList
isOdd(ns).result /* works fine! */
isEven(ns).result /* works fine! */

Optics in Scala

Lens explained

A short introduction to Lens in Scala, easily to understand is available here Based on this article, bellow is a short example ready to run.

object LensOptic extends App {
  case class Street(name: String, number: Int)
  case class Address(country: String, city: String, street: Street)

  case class Lens[A, B](
      get: A => B,
      set: (A, B) => A
  )

  def compose[Outer, Inner, Value](
      outer: Lens[Outer, Inner],
      inner: Lens[Inner, Value]
  ) =
    Lens[Outer, Value](
      get = outer.get andThen inner.get,
      set = (obj, value) => outer.set(obj, inner.set(outer.get(obj), value))
    )

  val addressStreetLens = Lens[Address, Street](
    get = _.street,
    set = (a, b) => a.copy(street = b)
  )

  val streetNumberLens = Lens[Street, Int](
    get = _.number,
    set = (a, b) => a.copy(number = b)
  )

  val bigStreet = Street("unirii", 3)
  val bigAddress = Address("ro", "sibiu", bigStreet)

  val addressStreetNumberLens: Lens[Address, Int] =
    compose(addressStreetLens, streetNumberLens)

  println(addressStreetNumberLens.set(bigAddress, 19))
  // Address(ro,sibiu,Street(unirii,19))
}

Monocle

Also you can use the Monocle library, an implemetation of Optics in Scala.

/** Same example using Monocle library */
import monocle.syntax.all._

  println(bigAddress.focus(_.street.number).modify(_ => 77))
  // Address(ro,sibiu,Street(unirii,77))

Play framework

Play Framework makes it easy to build web applications with Java & Scala. Play is based on a lightweight, stateless, web-friendly architecture. Built on Akka, Play provides predictable and minimal resource consumption (CPU, memory, threads) for highly-scalable applications.

Facebook OAuth2 integrartion.

object OAuth2Facebook {

  /** Request timeout */
  private val requestTimeout = 1000.millis

  /** API version */
  private val apiVersion = "v10.0"

  /**
    * Facebook OAuth2 integration
    *
    * @param ws          WSClient instance
    * @param accessToken Access token
    * @param userId      User ID
    * @return User details
    */
  def apply(ws: WSClient, accessToken: String, userId: String):
  Future[Either[Throwable, Map[String, Option[String]]]] = (
    for {
      valid <- userProfile(ws, accessToken)
      picture <- profilePicture(ws, userId) if valid.status == Status.OK
    } yield {
      val json = valid.json
      val pict = if (picture.status == Status.OK) Option(picture.underlying[Response].getUri.toString) else None
      Right(Map("user" -> (json \ "id").asOpt[String],
        "name" -> (json \ "name").asOpt[String],
        "email" -> (json \ "email").asOpt[String],
        "gender" -> (json \ "gender").asOpt[String],
        "picture" -> pict,
        "dob" -> (json \ "birthday").asOpt[String]))
    }) recover {
    case t: Throwable => Left(t)
  }

  /**
    * Retrieve user profile
    *
    * @param ws          WSClient instance
    * @param accessToken Access token
    */
  private def userProfile(ws: WSClient, accessToken: String) = {
    val fields = "id,name,email,picture,gender,birthday"
    ws.url(s"https://graph.facebook.com/$apiVersion/me")
      .withRequestTimeout(requestTimeout)
      .addQueryStringParameters("access_token" -> accessToken)
      .addQueryStringParameters("fields" -> fields)
      .get()
  }

  /**
    * Get user profile large picture.
    *
    * @param ws     WSClient instance
    * @param userId Facebook user ID
    */
  private def profilePicture(ws: WSClient, userId: String) = {
    ws.url(s"https://graph.facebook.com/$apiVersion/$userId/picture")
      .withRequestTimeout(requestTimeout)
      .addQueryStringParameters("type" -> "large")
      .get()
  }
}

Google OAuth2 integration.

object OAuth2Google {

  /** Request timeout */
  private val requestTimeout = 10000.millis

  /**
    * Google OAuth2 integration
    *
    * @param ws          WSClient instance
    * @param accessToken Google access token.
    * @param idToken     Google id token.
    * @return User details
    */
  def apply(ws: WSClient, accessToken: String, idToken: String):
  Future[Either[Throwable, Map[String, Option[String]]]] = (
    for {
      valid <- validate(ws, idToken)
      userinfo <- userInfo(ws, accessToken) if valid.status == Status.OK
      date <- dayOfBirth(ws, accessToken, (userinfo.json \ "sub").as[String]) if userinfo.status == Status.OK
    } yield {
      val birthday = if (date.status == Status.OK) (date.json \ "birthday").asOpt[String] else None
      val json = userinfo.json
      Right(Map("user" -> (json \ "sub").asOpt[String],
        "name" -> (json \ "name").asOpt[String],
        "email" -> (json \ "email").asOpt[String],
        "gender" -> (json \ "gender").asOpt[String],
        "picture" -> (json \ "picture").asOpt[String],
        "dob" -> birthday))
    }) recover {
    case t: Throwable => Left(t)
  }

  /**
    * Validate token.
    *
    * @param ws      WSClient instance
    * @param idToken Google id token.
    */
  private def validate(ws: WSClient, idToken: String) = {
    ws.url("https://www.googleapis.com/oauth2/v3/tokeninfo")
      .withRequestTimeout(requestTimeout)
      .addQueryStringParameters("id_token" -> idToken)
      .get()
  }

  /**
    * Get user details. (information)
    *
    * @param ws          WSClient instance
    * @param accessToken Access token.
    */
  private def userInfo(ws: WSClient, accessToken: String) =
    ws.url("https://www.googleapis.com/oauth2/v3/userinfo")
      .withRequestTimeout(requestTimeout)
      .addQueryStringParameters("alt" -> "json")
      .addQueryStringParameters("access_token" -> accessToken)
      .get()

  /**
    * Google birthday is accessible only if the user made public the information.
    * The birthday format in JSON response is YYYY-MM-DD
    * Also a Google user could hide the year of birth. The year in response is substituted with 0000 and a
    * typical response in this case looks like 0000-MM-DD
    *
    * @param ws          WSClient instance
    * @param accessToken Access token
    * @param personId    Person ID
    */
  private def dayOfBirth(ws: WSClient, accessToken: String, personId: String) =
    ws.url(s"https://www.googleapis.com/plus/v3/people/$personId")
      .withRequestTimeout(requestTimeout)
      .withHttpHeaders(HeaderNames.AUTHORIZATION -> s"Bearer $accessToken")
      .get()
}

GitHub links

Below are some links to my GitHub repositories where you can find some Scala solutions from our implementations.

Haskell, an advanced, purely FP language

Haskell is an advanced, purely functional programming language. It is a reference when it comes to functional programming.

Recursion and pattern matching.

Haskell does not have an intrinsic equivalent of while loops based on mutable state. However, using recursion and pattern matching it is possible to substitute imperative for loops. Below are some examples.

-- compute the length of a list
length :: [a] -> Int
length [] = 0
length (x:xs) = 1 + length xs

-- quick sort
quickSort :: [Int] -> [Int]
quickSort [] = []
quickSort (x:xs) = left ++ [x] ++ right
    where
        left = quickSort $ filter (<=x) xs
        right = quickSort $ filter (>x) xs

-- factorial computation
fact :: Integer -> Integer
fact 0 = 1
fact n = n * fact(n - 1)

-- fibonacci computation
fib :: Int -> Int
fib 0 = 0
fib 1 = 1
fib n = fib (n - 1) + fib (n - 2)

Heskell type classes

In Haskell, type classes provide a structured way to control ad hoc polymorphism, or overloading.

class Semigroup m where
  (<>) :: m -> m -> m

class Semigroup m => Monoid m where
  mempty :: m

class Functor f where
  fmap :: ( a -> b ) -> f a -> f b

class Functor f => Applicative f where
  pure  :: a -> f a
  (<*>) :: f (a -> b) -> f a -> f b

class Monad m where
  (>>=)  :: m a -> (a -> m b) -> m b
  (>>)   :: m a ->  m b       -> m b
  return ::   a               -> m a
  fail   :: String -> m a

class Show a where
  show :: a -> String

class Eq a  where
  (==), (/=) :: a -> a -> Bool

class (Eq a) => Ord a where
  compare              :: a -> a -> Ordering
  (<), (<=), (>=), (>) :: a -> a -> Bool
  max, min             :: a -> a -> a

class  Enum a  where
  succ, pred     :: a -> a
  toEnum         :: Int -> a
  fromEnum       :: a -> Int
  enumFrom       :: a -> [a]            -- [n..]  
  enumFromThen   :: a -> a -> [a]       -- [n,n'..]  
  enumFromTo     :: a -> a -> [a]       -- [n..m]  
  enumFromThenTo :: a -> a -> a -> [a]  -- [n,n'..m]  

class Num a where
  (+) :: a -> a -> a
  (-) :: a -> a -> a
  (*) :: a -> a -> a
  negate :: a -> a
  abs :: a -> a
  signum :: a -> a

class Monad m => MonadReader r m | m -> r where
  ask    :: m r                    -- retrieves the monad environment
  local  :: (r -> r) -> m a -> m a -- executes a computation in a modified environment
  reader :: (r -> a) -> m a        -- retrieves a function of the current environment

class  Bounded a  where
  minBound, maxBound :: a

data Maybe a  =  Nothing | Just a
            (Eq, Ord)

instance Monad Maybe where
  return = Just
  Nothing >>= f = Nothing
  (Just x) >>= f = f x
  fail _ = Nothing

data Bool  =  False | True deriving
            (Eq, Ord, Bounded, Enum, Read, Show)

data Ordering  =  LT | EQ | GT deriving
            (Eq, Ord, Bounded, Enum, Read, Show)

Lambda abstraction

Instead of using equations to define functions, we can also define them "anonymously" via a lambda abstraction.

inc x = x+1
add x y = x+y
-- are really shorthand for:
inc = \x -> x+1
add = \x y -> x+y

Lambda calculus

λ-calculus is a formal notation of a function from computational perspective.

ID := λx.x
INC := λx.x+1
ADD := λx.λy.x+y
INC := λx.ADD 1

TRUE := λx.λy.x
FALSE := λx.λy.y
NAND := λx.λy.x (y FALSE TRUE) TRUE
NOT := λx. NAND x x
AND := λx. λy. NOT (NAND x y)
OR := λx. λy. NAND (NOT x) (NOT x)

-- Haskell Curry Y combinator
Y := λf. (λx. f (x x)) (λx. f (x x))

Java is a solid platform to design web applications

Spring Boot 2.x, Spring Cloud microservices PoC

A microservices architecture using Spring Boot 2.x, Spring Cloud Netflix Hoxton.SRx, Keycloak Identity and Access Management and ELK stack. All components are running inside Docker containers.

Below is a code snippet from the UI Authorization Controller, an OAuth2 implementation of Authorization Code Grant Type. The Authorization Code Grant Type is the most commonly used grant type to authorize the Client to access protected data from a Resource Server.

@GetMapping(value = "/authorize")
public RedirectView redirectWithUsingRedirectView(RedirectAttributes attributes) {
    String state = stateBuilder();
    UriComponentsBuilder builder = UriComponentsBuilder.fromHttpUrl(auth)
            .queryParam("response_type", "code")
            .queryParam("client_id", client)
            .queryParam("redirect_uri", redirect)
            .queryParam("scope", scope)
            .queryParam("state", state);

    attributes.addFlashAttribute("state", state);
    return new RedirectView(builder.toUriString());
}

@GetMapping(value = "/authorized")        // registered redirect_uri for authorization_code
public String authorized(@RequestParam String code,
                         @RequestParam String state,
                         @RequestParam String session_state,
                         @ModelAttribute("state") String stateInit) {
    if (state.equals(stateInit)) {
        // params
        MultiValueMap<String, String> params = new LinkedMultiValueMap<>();
        params.set("grant_type", "authorization_code");
        params.set("client_id", client);
        params.set("client_secret", secret);
        params.set("code", code);
        params.set("scope", scope);
        params.set("redirect_uri", redirect);
        // headers
        MultiValueMap<String, String> headers = new LinkedMultiValueMap<>();
        headers.set(HttpHeaders.ACCEPT, MediaType.APPLICATION_JSON_VALUE);
        headers.set(HttpHeaders.CONTENT_TYPE, MediaType.APPLICATION_FORM_URLENCODED_VALUE);
        // call
        HttpEntity<MultiValueMap<String, String>> httpEntity = new HttpEntity<>(params, headers);
        ResponseEntity<String> response = restClient.exchange(uri, HttpMethod.POST, httpEntity, String.class);
        // send back response
        return storeToken(response) ? "orders" : "index";
    } else {
        return "index";
    }
}

RTSP/RTP torrent

Is a small and fast RTSP/RTP client Java library used to get streaming data from IP cameras or streaming servers. RTSP client is RFC 2326 compliant and RTP client is RFC 3550 compliant.

RtpVideoPlayer videoPlayer = new RtpVideoPlayer("0.0.0.0", 9000);
RtpAudioPlayer audioPlayer = new RtpAudioPlayer("0.0.0.0", 9002);

RtspSession session = new RtspSession(videoPlayer, audioPlayer);
session.setUsername("username");
session.setPassword("password");

session.play("rtsp://hostname:554/stream");

Database Stored Procedures ORM

Abstract

Today almost all Java applications having persistence layers are designed using the JPA (Java Persistence API) and/or one of the existing ORM frameworks like Hibernate, EclipseLink, etc. In Hibernate Reference Manual preface we are advised that Hibernate may not be the best solution when we have to work with stored procedures. "Hibernate may not be the best solution for data-centric applications that only use stored-procedures to implement the business logic in the database, it is most useful with object-oriented domain models and business logic in the Java-based middle-tier."

Solution

If you have to call stored procedures in your applications, you have to use the standard JDBC API working with CallableStatement objects. But now you can use a more simple and efficient solution using an annotated POJO and a ProcedureManager instance. Designed from scratch the pojo-sp library offers the best object-oriented (OOP) solution to call stored procedures from your applications by hiding the JDBC SQL programming artefacts. For each stored procedure or function call you have to create a POJO, a simple Java Bean class with annotations. The POJO class MUST be decorated with a @StoredProcedure annotation and you have to specify the name of the stored procedure or function that is mapped by your class. If the procedure or the function is from an Oracle package you have to specify also the name of the package: PACKAGE_NAME.PROCEDURE_NAME Next you have to specify if the entity you are calling is a procedure or a function. This attribute is by default true for stored procedures and MUST be set to false if the entity you are calling is a function. The difference between a procedure and a function is that a function always returns a value. Inside your POJO class you have to define the stored procedure parameters call and decorate the fields with @StoredProcedureParameter annotation. You have to specify the index of your parameters starting from 1 for the first parameter. The names of the parameters are not important because the parameters are accessed by index not by names. If the entity is a function the first parameter (index = 1) is always the return value. The next attribute is the SQL type of the parameter. Here you have to be carefully because this is a mapping of an SQL type to a Java type and this mapping MUST match. The last attribute is the direction for this parameter which can be one of the IN, OUT or INOUT value. For the result parameter of a function you MUST specify always OUT as a direction attribute.

@StoredProcedure(name = "HELLO", procedure = false)
public class Hello {

  @StoredProcedureParameter(index = 1, type = Types.VARCHAR, direction = Direction.OUT)
  private String result;

  @StoredProcedureParameter(index = 2, type = Types.VARCHAR, direction = Direction.IN)
  private String name;

  public String getName() {
    return name;
  }

  public void setName(String name) {
    this.name = name;
  }

  public String getResult() {
    return result;
  }

  public void setResult(String result) {
    this.result = result;
  }
}

The final step is to create a ProcedureManager instance, set the input parameters in your POJO class and call the stored procedure. The instance is created using the ProcedureManagerFactory.createInstance() method, using a DataSource, a JDBC Connection object as parameter or a class decorated with @JDBC annotation. If you need to use transactions a TransactionManager interface is available to use. Also, all "checked" exceptions are converted to "unchecked" exceptions, and it's not mandatory to use a try catch block in your code.

@JDBC(driver = "oracle.jdbc.OracleDriver",
  url = "jdbc:oracle:thin:@127.0.0.1:1521:XE",
  username = "HR", password = "hr")
public class Main {

  public void test() {
    ProcedureManager pm = ProcedureManagerFactory.createInstance(Main.class);
    try {
      pm.getTransactionManager().begin();

      // Call HELLO function
      Hello hello = new Hello();
      hello.setName("Marius");
      pm.call(hello);
      System.out.println(hello.getResult());

      pm.getTransactionManager().commit();
    } catch (Exception e) {
      pm.getTransactionManager().rollback();
    } finally {
      pm.close();
    }
}

DEC VT100 emulator

JNI native code can be used to implement a VT100 terminal emulator which is running as a Java Swing standalone application.

GitHub links

Below are some links to my GitHub repositories where you can find some Java solutions from our implementations.

Python is widely used nowadays, it became a must

Microsoft Outlook MSG parser

Parse Microsoft Outlook messages and extract properties like recipients, attachments and many others. It is a complete Python implementation of Microsoft [MS-CFB]: Compound File Binary File Format specifications. Below is a code snippet of CFB directory implementation.

class CfbDirectory:
    __slots__ = ['entries']

    def __init__(self, _data):
        self.entries = _data
        assert (_data[0].object_type() == OBJ_TYPE_ROOT_STORAGE)
        logger.log.debug("directory entries: %d" % len(_data))
        for i in range(0, len(self.entries)):
            self.entries[i].index = i
            self.entries[i].info()
        self._make_tree()

    def _make_tree(self):
        logger.log.debug("make tree")
        for entry in self.entries:
            if entry.object_type() in (OBJ_TYPE_STORAGE, OBJ_TYPE_ROOT_STORAGE):
                entry.set_children(self.add_children(entry))

    def add_children(self, entry):
        children = []
        if 0 <= entry.child_id() <= MAXREGSECT:
            children.append(entry.child_id())
            child = self.entries[entry.child_id()]
            self.traverse(child, children)
        return sorted(children)

    def traverse(self, child, children):
        if child.left_sibling() != FREESECT:
            children.append(child.left_sibling())
            self.traverse(self.entries[child.left_sibling()], children)

        if child.right_sibling() != FREESECT:
            children.append(child.right_sibling())
            self.traverse(self.entries[child.right_sibling()], children)

    def find_entry_by_name(self, name):
        root = self.entries[0]
        assert (root.object_type() == OBJ_TYPE_ROOT_STORAGE)
        for child in root.children:
            if self.entries[child].directory_entry_name() == name:
                return self.entries[child]
        return None

    def select_entry_by_name(self, name):
        group = []
        root = self.entries[0]
        assert (root.object_type() == OBJ_TYPE_ROOT_STORAGE)
        for child in root.children:
            if self.entries[child].directory_entry_name().startswith(name):
                group.append(self.entries[child])
        return group

    def root(self):
        return self.entries[0]

    def entry(self, index):
        assert (0 <= index < len(self.entries))
        return self.entries[index]

    def debug(self, name):
        entry = self.find_entry_by_name(name)
        entry.info()
        for child in entry.children:
            self.entries[child].info()

    def all(self):
        for entry in self.entries:
            entry.info()

SSH Client

Using SSH in Python is easier than ever with Paramiko package. Below is a Python implementation of a SSH client using paramiko package.

class SSHClient:
    __slots__ = ['host', 'port', 'username', 'password', 'client']
    """
    Simple SSH client.
    Example:

        with SSHClient("192.168.1.56", 22, "admin", "admin") as cli:
            stdin, stdout, stderr = cli.shell("ls -la")
            for line in stdout:
                print(line.strip('\n'))
    """

    def __init__(self, host='localhost', port=22, username=None, password=None):
        """
        Constructor

        :param host: host name or ip address
        :param port: port number
        :param username: username
        :param password: user password
        """
        self.host = host
        self.port = port
        self.username = username
        self.password = password

    def __enter__(self):
        self.client = paramiko.SSHClient()
        self.client.set_missing_host_key_policy(paramiko.AutoAddPolicy())
        self.client.connect(self.host, self.port, self.username, self.password)
        return self

    def __exit__(self, exception_type, exception_value, traceback):
        self.client.close()

    def shell(self, command):
        """
        Execute a command on the remote machine.

        :param command: command line
        :return: stdin, stdout, stderr objects of remote machines
        """
        shell = self.client.invoke_shell()
        stdin, stdout, stderr = self.client.exec_command(command)
        shell.close()
        return stdin, stdout, stderr

    def upload_file(self, src, dst):
        """
        Upload a file to remote host.

        :param src: source file
        :param dst: destination file
        :return: Nothing
        """
        sftp = self.client.open_sftp()
        sftp.put(src, dst)
        sftp.close()

    def download_file(self, src, dst):
        """
        Download a file from remote host

        :param src: source file
        :param dst: destination file
        :return: Nothing
        """
        sftp = self.client.open_sftp()
        sftp.get(src, dst)
        sftp.close()

Cassandra reading partition and clustering keys column names

Cassandra is a NoSQL distributed database in which data is partitioned and stored across multiple nodes within a cluster. The partition key is made up of one or more data fields and is used by the partitioner to generate a token via hashing to distribute the data uniformly across a cluster. A clustering key is made up of one or more fields and helps in clustering or grouping together rows with same partition key and storing them in sorted order. The combination of partition key and clustering key makes up the primary key and uniquely identifies any record in the Cassandra cluster.

def get_table_columns(session, key_space, table_name):
  pk = {}
  cc = {}
  rc = []
  query = "select * from system_schema.columns where keyspace_name=%s and table_name=%s"
  rows = session.execute(query, parameters=[key_space, table_name], timeout=None)
  for row in rows:
      if row.kind == "partition_key":
          pk[row.position] = row.column_name, row.type
      elif row.kind == "clustering":
          cc[row.position] = row.column_name, row.type, row.clustering_order
      elif row.kind == "regular":
          rc.append((row.column_name, row.type))

  return [pk[key] for key in sorted(pk.keys())],\
          [cc[key] for key in sorted(cc.keys())], sorted(rc, key=lambda x: x[0])

Expression evaluator

Python AST (Abstract Syntax Tree) module helps us to design an interesting expression evaluator.

#!/usr/bin/python
# -*- coding: utf-8 -*-
__author__ = 'Marius'

import ast


def eval_node(node, params):
    """
    Evaluate node
    """

    if isinstance(node, ast.Num):
        return ast.literal_eval(node)
    elif isinstance(node, ast.Str):
        return ast.literal_eval(node)
    elif isinstance(node, ast.Name):
        return params[node.__dict__['id']]
    elif isinstance(node, ast.BinOp):
        return eval_bin_node(node, params)
    elif isinstance(node, ast.UnaryOp):
        return eval_unary_node(node, params)
    elif isinstance(node, ast.BoolOp):
        return eval_bool_node(node, params)
    elif isinstance(node, ast.Compare):
        return eval_comp_node(node, params)
    else:
        raise Exception('Unsupported node type.')


def eval_bin_node(node, params):
    """
    Evaluate BinOp node.
    """

    # evaluate left and right nodes
    left_value = eval_node(node.__dict__['left'], params)
    right_value = eval_node(node.__dict__['right'], params)

    # operator
    operator = node.__dict__['op']

    # Arithmetic Operators
    if isinstance(operator, ast.Add):
        return left_value + right_value
    if isinstance(operator, ast.Sub):
        return left_value - right_value
    if isinstance(operator, ast.Mult):
        return left_value * right_value
    if isinstance(operator, ast.Div):
        return left_value / right_value
    if isinstance(operator, ast.Mod):
        return left_value % right_value
    if isinstance(operator, ast.Pow):
        return left_value ** right_value
    if isinstance(operator, ast.FloorDiv):
        return left_value // right_value

    # Bitwise Operators
    if isinstance(operator, ast.BitOr):
        return left_value | right_value
    if isinstance(operator, ast.BitAnd):
        return left_value & right_value
    if isinstance(operator, ast.BitXor):
        return left_value ^ right_value
    if isinstance(operator, ast.LShift):
        return left_value << right_value
    if isinstance(operator, ast.RShift):
        return left_value >> right_value
    else:
        raise Exception('Unsupported operator.')


def eval_comp_node(node, params):

    # left operand
    left_value = eval_node(node.__dict__['left'], params)

    # operator
    operators = node.__dict__['ops']

    # right operators
    comparators = node.__dict__['comparators']
    for i in range(len(operators)):

        # evaluate right node
        right_value = eval_node(comparators[i], params)

        # operator
        operator = operators[i]

        # Comparison (i.e., Relational) Operators
        if isinstance(operator, ast.Eq):
            left_value = left_value == right_value
        elif isinstance(operator, ast.NotEq):
            left_value = left_value != right_value
        elif isinstance(operator, ast.Lt):
            left_value = left_value < right_value
        elif isinstance(operator, ast.LtE):
            left_value = left_value <= right_value
        elif isinstance(operator, ast.Gt):
            left_value = left_value > right_value
        elif isinstance(operator, ast.GtE):
            left_value = left_value >= right_value
        else:
            raise Exception('Unsupported operator')

    return left_value


def eval_bool_node(node, params):
    """
    Evaluate boolean expressions
    """

    left_value = None

    # operator
    operator = node.__dict__['op']

    for right in node.__dict__['values']:
        right_value = eval_node(right, params)
        if left_value is None:
            left_value = right_value
        else:

            # Boolean Operators
            if isinstance(operator, ast.And):
                left_value = left_value and right_value
            elif isinstance(operator, ast.Or):
                left_value = left_value or right_value
            else:
                raise Exception('Unknown operator.')

    return left_value


def eval_unary_node(node, params):
    """
    Evaluate unary expressions
    """

    # Boolean Operators
    operator = node.__dict__['op']
    if isinstance(operator, ast.Not):
        return not eval_node(node.__dict__['operand'], params)
    else:
        raise Exception('Unknown operator.')


def eval_expr(expression, params):
    """
    Evaluate an expression using python ast (abstract syntax tree)
    """

    tree = ast.parse(expression)

    # module
    module = ast.walk(tree).next()
    if not isinstance(module, ast.Module):
        raise Exception('Missing ast.Module node.')

    # print ast.dump(module, True)
    expr_node = module.__dict__['body'][0]
    if not isinstance(expr_node, ast.Expr):
        raise Exception('Missing ast.Expr node.')

    # evaluate expression
    expr_value = expr_node.__dict__['value']
    if isinstance(expr_value, ast.BinOp):
        return eval_bin_node(expr_value, params)
    if isinstance(expr_value, ast.UnaryOp):
        return eval_unary_node(expr_value, params)
    if isinstance(expr_value, ast.BoolOp):
        return eval_bool_node(expr_value, params)
    elif isinstance(expr_value, ast.Compare):
        return eval_comp_node(expr_value, params)
    else:
        print expr_value
        raise Exception('Unsupported node type.')


if __name__ == '__main__':
    formula = '2.67 + (1 + data / 1000.0 + data**2)'
    print eval_expr(formula, {'data': 314})
    print eval_expr('x and y', {'x': True, 'y': True})
    print eval_expr('not x', {'x': True})

Add a GZIP filter to Falcon

Falcon is a minimalist WSGI library for building speedy web API's and app backends. We are using Falcon to build REST API's. The code snippet below has been designed to enhance the framework with a GZIP compression filter.

class GzipMiddleware(object):
    """GZIP compression middleware

    """
    __slots__ = ('compression_min_size', 'compress_level')

    def __init__(self, compression_min_size=2048, compress_level=5):
        self.compression_min_size = compression_min_size
        self.compress_level = compress_level

    def process_response(self, req, resp, resource):
        """Post-processing of the response (after routing).

        Args:
            req: Request object.
            resp: Response object.
            resource: Resource object to which the request was
                routed. Maybe None if no route was found
                for the request.
        """
        headers = req.headers
        accept_hdr = headers.get('ACCEPT-ENCODING')
        if accept_hdr and 'gzip' in accept_hdr.lower():
            data = resp.body
            if data and len(data) >= self.compression_min_size:
                gzip_stream = io.BytesIO()
                with gzip.GzipFile(fileobj=gzip_stream, mode="w", compresslevel=self.compress_level) as f:
                    f.write(bytes(data, "utf-8"))
                resp.data = gzip_stream.getvalue()
                resp.body = None
                resp.append_header('Content-Encoding', 'gzip')

GitHub links

Below are some links to my GitHub repositories where you can find some Python solutions from our implementations.

Microsoft Outlook MSG parser

JavaScript

It's almost impossible to be a Software Developer these days without using JavaScript in some way!

Pure JavaScript Motion JPEG player

Using JavaScript and HTML5 canvas, it is possible to write miscellaneous functions like this amazing Motion JPEG player.

function play(renderer, url, sleep) {
  var can = document.getElementById(renderer);
  var ctx = can.getContext('2d');
  ctx.fillStyle = 'black';
  ctx.fillRect(0, 0, can.width, can.height);
  window.setInterval(function(){
      var img = new Image();
      img.onload = function() {
          ctx.drawImage(img, 0, 0, can.width, can.height);
          img = null;
      };
      var rand = '?rand=' + new Date().getTime();
      img.src = url + rand;
  }, sleep);
}

What's Ajax

Ajax stands for Asynchronous Javascript And Xml. Ajax is just a means of loading data from the server and selectively updating parts of a web page without reloading the whole page. Basically, what Ajax does is make use of the browser's built-in XMLHttpRequest (XHR) object to send and receive information to and from a web server asynchronously, in the background, without blocking the page or interfering with the user's experience. Ajax has become so popular that you hardly find an application that doesn't use Ajax to some extent. The example of some large-scale Ajax-driven online applications are: Gmail, Google Maps, Google Docs, YouTube, Facebook, Flickr, and so many other applications.

Native JavaScript Ajax module example:

var $ajax = (function () {

  return {
    call: function (uri, type, options) {
      /* parse options */
      var async = (options.async) ? options.async : true;
      var data = (options.data) ? options.data : null;
      var user = (options.user) ? options.user : null;
      var password = (options.password) ? options.password : null;
      var headers = (options.headers) ? options.headers : new Map();

      var xmlhttp = (window.XMLHttpRequest) ?
        new XMLHttpRequest() : new ActiveXObject('Microsoft.XMLHTTP');

      xmlhttp.onerror = function () {
        if (options.error)
          options.error(xmlhttp);
      }

      xmlhttp.onreadystatechange = function () {
        if (xmlhttp.readyState == 4 && xmlhttp.status == 200) {
          if (options.success)
            options.success(xmlhttp);
        } else {
          if (options.error)
            options.error(xmlhttp);
        }
      }

      xmlhttp.open(type, uri, async, user, password);

      /* set request headers */
      for (const [key, value] of headers) {
        xmlhttp.setRequestHeader(key, value);
      }

      xmlhttp.send(data);
    }
  }
}());

Ajax GET request example:

$ajax.call('https://reqres.in/api/unknown', 'GET', {
  'headers': new Map([
    ['Cache-Control', 'max-age=0'],
    ['Authorization', 'Bearer token']
  ]),
  'success': function (xmlhttp) {
    console.log(xmlhttp);
  },
  'error': function (xmlhttp) {
    console.log(xmlhttp);
  }
});

Ajax POST request example:

$ajax.call('https://reqres.in/api/login', 'POST', {
  'headers': new Map([
    ['Content-Type', 'application/json']
  ]),
  'data': JSON.stringify({
    "email": "eve.holt@reqres.in",
    "password": "cityslicka"
  }),
  'success': function (xmlhttp) {
    console.log(xmlhttp);
  },
  'error': function (xmlhttp) {
    console.log(xmlhttp);
  }
});

Test Ajax JavaScript module

JavaScript module design pattern

The Module Pattern is one of the important patterns in JavaScript. It's used to wrap a set of variables and functions together in a single scope.

MODULE = (function ($) {
  var my = {};
  var privateVariable = 'jQuery version ';

  function privateMethod() {
    // ...
  }

  my.moduleProperty = 1;
  my.moduleMethod = function () {
    alert(privateVariable + $().jquery)
    // ...
  }

  return my;
}(jQuery));

MODULE.moduleMethod()
console.log(MODULE.moduleProperty)

Linux terminal running in your web browser.

JavaScript with WebSockets is used to implement a Linux terminal embedded in a web page. A Java VT100 emulator is running in the backend.

C++ is an extension of C programming language

Using C and C++ it's possible to design and implements high-tech applications like video and audio capture, compressing and streaming. Below is a code snippet from a DirectShow application which enums all the video and audio capture devices available on the host machine.

void CDevice::GetSources(HWND hWnd, REFCLSID category)
{
  HRESULT hr;
  // Create the System Device Enumerator.
  ICreateDevEnum *pSysDevEnum;
  hr = CoCreateInstance(CLSID_SystemDeviceEnum, NULL, CLSCTX_INPROC_SERVER,
                        IID_ICreateDevEnum, (void **)&pSysDevEnum);
  if (hr != S_OK)
  {
    return;
  }

  // Create an enumerator for a specified device category.
  IEnumMoniker *pEnumCat;
  hr = pSysDevEnum->CreateClassEnumerator(category, &pEnumCat, 0);
  if (hr != S_OK)
  {
    hr = pSysDevEnum->Release();
    return;
  }

  // Enumerate the monikers.
  ULONG cFetched = 0;
  IPropertyBag *pPropBag = 0;
  IMoniker *pMoniker = 0;
  VARIANT varName;

  while (pEnumCat->Next(1, &pMoniker, &cFetched) == S_OK)
  {
    if (pMoniker->BindToStorage(0, 0, IID_IPropertyBag, (void **)&pPropBag) == S_OK)
    {
      // To retrieve the friendly name of the filter, do the following:
      VariantInit(&varName);
      hr = pPropBag->Read(L"FriendlyName", &varName, 0);
      if (hr == S_OK)
      {
        ::SendMessageW(hWnd, CB_ADDSTRING, 0, (LONG)varName.bstrVal);
      }
      VariantClear(&varName);
      hr = pPropBag->Release();
    }
    hr = pMoniker->Release();
  }

  // clean up
  hr = pEnumCat->Release();
  hr = pSysDevEnum->Release();
}

Old C++ applications

Downloads

Below are some links to my C, C++ free applications.

Contact

For any question or information, please do not hesitate to contact me on the following channels.