Setup

This is a continuation on the series where we walk through creating an Angular, DotNet Core microservice project.

Download links for the software mentioned in this post will be available at the bottom, although this will not be maintained.

How you choose to setup your development machine will differ depending on which platform you intend to develop on (Windows, Linux or OSX). I am going to be using Windows 10 Pro for my development machine, however, I will attempt to include steps/scripts for following the same steps on a Debian Linux box, and eventually OSX, where applicable.

You may wonder why I don’t just stick to using Windows or Debian, well that is a good question. The answer is that I am going to target one or more Debian instances later on; building up some knowledge for both might be beneficial later on.

Update your OS

Whether you are using Windows or Linux etc. you will want to make sure that your machine is up-to date with the latest stable release. This is to ensure that any security exploits that may be out there do not affect you when downloading setup files etc. from the internet.

In Windows:

Go to “Settings”, “Update & Security” and click on “Check for updates”. Your machine should then check for updates and inform you of the steps involved to obtain and install them. If you are on a corporate network this may be controlled by your local IT team.

On Debian:

Run the following command:
$ sudo apt-get update && apt-get upgrade
This will update the package lists and ask if you want to install the available updates.

Install development applications

Here is the Manual process to download and install the following applications, I have included a Windows Batch/PowerShell script which will automate some of the process for you.

Visual Studio Code

Visit https://code.visualstudio.com/ and click on the “Download for Windows” link.

I’m going to select all of the options, as I am using a clean machine. You may want to customize depending on your existing setup.

VS Code will prompt you to download and install Git, if it’s not already installed, we’ve already downloaded Git so you can ignore this message.

Git

Visit https://git-scm.com/download/win download the latest version, alternatively just use the following download link https://github.com/git-for-windows/git/releases/download/v2.16.2.windows.1/Git-2.16.2-64-bit.exe (as of March 2018).

Once the file has downloaded run the executable to install Git.

If VS Code isn’t already installed, you won’t be able to select the Use Visual Studio Code option.

When VS Code is installed you can select this option and proceed.

Select Unix-style line endings, this makes things a lot easier when working with the Debian instances later.

I’m going to select the Windows default console window, mostly because I’m old fashioned also because it works with NodeJS.

7Zip

If you want to build OpenSSL manually I would recommend installing 7Zip. It is generally quite useful for use with Zip archives and a good alternative to the built in Windows functionality.

Visit http://www.7-zip.org and download the latest version, alternatively use the link below.

Once downloaded run the executable (.exe).

Node

Visit https://nodejs.org and download the latest version, alternatively use the link below.

OpenSSL

Visit https://slproweb.com/products/Win32OpenSSL.html and download the light/standard version of OpenSSL.

Once downloaded run the executable (.exe).

DotNet Core SDK

As we’re going to be building some dotnet Core WebAPI applications we are going to require the dotnet Core SDK.

Visit https://www.microsoft.com/net/learn/get-started/windows and click on the “Download .NET SDK” link.

Once downloaded run the executable (.exe)

nGinx

Visit http://nginx.org/en/download.html and download the latest Zipped version.

Don’t do anything with the file yet, we will need it later.

PHP

Visit https://windows.php.net/download#php-7.2 and download the latest VC15 x64 Thread Safe version of the Zip file.

On my machine I have downloaded the Zip archive from https://windows.php.net/downloads/releases/php-7.2.3-Win32-VC15-x64.zip

Don’t do anything with the file yet, we will extract it later.

Visual C++ Runtime 64bit

We need to this runtime in order to use PHP, as Fast CGI depends on it.

You can download this from https://www.microsoft.com/en-GB/download/details.aspx?id=48145, make sure you download the 64bit version.

OK, that’s it for now. The next step is to create our project folders.

Download links:

• Git – https://github.com/git-for-windows/git/releases/download/v2.16.2.windows.1/Git-2.16.2-64-bit.exe
• 7Zip – http://www.7-zip.org/a/7z1801.exe
• Node – https://nodejs.org/dist/v8.9.4/node-v8.9.4-x64.msi
• OpenSSL – https://slproweb.com/download/Win32OpenSSL_Light-1_1_0g.exe
• DotNet Core SDK – https://download.microsoft.com/download/1/1/5/115B762D-2B41-4AF3-9A63-92D9680B9409/dotnet-sdk-2.1.4-win-x64.exe
• VSCode – https://go.microsoft.com/fwlink/?Linkid=852157
• nGinx – http://nginx.org/download/nginx-1.13.9.zip
• PHP – https://windows.php.net/downloads/releases/php-7.2.3-Win32-VC15-x64.zip
• VC Runtime – https://download.microsoft.com/download/9/3/F/93FCF1E7-E6A4-478B-96E7-D4B285925B00/vc_redist.x64.exe

Technology

In my last post, I eluded to wanting to document the process of creating an Angular, DotNet Core microservice project.

Before I dive into the instructions I thought I would first offer some justifications as to which technologies I plan to use and where we might end up.

I have decided to challenge myself to learning the following technologies; as they represent a complete set of technology layers found in use today, they also provide a good method of deconstructing a project into parts, which can be fully de-coupled and interchanged:

API Development

VSCode

I have played with VSCode personally and at work, it is fast which makes it a complete dream to use. I can work with multiple languages in the same IDE and manage GIT changes etc. all within one place.

(Dot).Net Core

I am familiar with C# and the .Net framework so it seemed logical to target this first new branch of the technology, especially as Core can be used on a multitude of different environments including Linux, OSX & Windows.

PHP

I have used PHP in the past for personal projects; it is quick to script something together, although speed can completely depend on how complex the task is. I have previously used Slim and Twig for a basic MVC application so I might go down that route again, as opposed to using Laravel/Symphony. We’ll see how it goes.

Node.JS

I was going to say I have never used Node but that would be a lie. I have never developed anything to specifically run as a Node application. JavaScript is ubiquitous, and as I have found out, Node.JS is forging JavaScript’s path forward. It makes sense to at least investigate what can be achieved and compare it with dotnet Core and PHP.

Web Server/Reverse Proxy

nGinx

I have played around with nGinx at home, along with PHP and Core, to test how to reverse proxy Core MVC applications and server PHP pages. I plan to expand on my knowledge of nGinx configuration and leverage some tricks to build a single application out of many technologies but obfuscate it all through nGinx.

Application containerisation/hosting

Docker

Docker intrigues me, I’ve used virtual machines for over a decade now but have not yet used containers, I feel this is definitely something that is going to grow as time goes on and is going to be of paramount importance to software developers going forward. Docker also has the benefits that it can be used to scale all, or part, of my planned application. This is something that should be of interest to anyone planning to deploy a scalable application.

Debian

Debian is great, it is the basis of some of the most popular Linux distributions out there – end of. I’m going to attempt to deploy my complete application to Docker Debian containers on a Debian virtual server.

Source Control

GIT

Don’t panic, I’ve been using source control! My current experience has focused on basic usage of TFS in a corporate setting. For this project I want to expand my knowledge of GIT by making this project public via GitHub. I’ve used Git previously and I have used BitBucket and SourceTree but GIT is cross platform, which is how things should be.

HTTPS certificates

OpenSSL

I have been a fan of OpenSSL for a long time, I use it at work and will continue to do so for some time. We’re going to use it to produce some test certificates on our development environment.

LetsEncrypt

I have been using LetsEncrypt for about a year and cannot fault it for the price 😉. It is a great service and the developers have made the process as simple as possible to be able to create and renew certificates on a Linux system, I haven’t even tried using it on Windows.

At the time of writing, LetsEncrypt are on the verge of releasing wildcard certificates, however, I will cover how to create multiple subdomain certificates on the same nGinx server. If wildcard certificates become available, I will also include this within the guide.

Database

SQL

SQL and I go way back, about 15 years or so. Now it is available on Linux and as an Express addition – it should meet my needs.

SQLite

SQLite is a quick win when it comes to designing a database structure and testing it out. I may still use this during development, although if I can quickly spin up a SQL instance in Docker the benefits are slightly diminished.

MySQL

I’m not sure if SQL will allow me to do all of the High Availability stuff that I believe I can do with MySQL in a Docker swarm. Time will tell if I need to switch from MSSQL to MySQL.

Why are you (not) using…?

Good question(s). This document focuses on constructing an application for deployment on machines that I have complete control of. Although it can be potentially torturous, I believe it is good practice to occasionally push yourself to understand, as much as possible, the infrastructure that your applications are running on. This provides a deeper knowledge which in-turn aides you to diagnose issues when things go wrong.

I’ve shied away from using Azure and AWS for hosting, as I want to try to understand as much of the stack as possible. This also frees me from the pressures of any cost implications, which as a parent of small children is a blessing – spare time and money are not something I have in abundance.

Perhaps later on, when I have completed the initial version of the document, I will be able to expand further into various hosting scenarios and other configurations.

Configuration

Question: I don’t have access to a Windows 10 Pro machine, can I still use this guide?

In short, yes. I’m initially writing this document for myself whilst using Windows 10 Pro. I do however have a paunchent for running Linux machines for my other personal projects.

The stack of technologies I have employed for this project are available across multiple OSs so in theory nothing should stop you from following this guide on a Linux or OSX machine. The only thing that will not work for you are some of the scripts and configuration files, I will aim to create an alternative set of these files for other OSs.

New Series of posts

Sorry

OK, so I have been pretty quiet for a while. I’ve been struggling to juggle my time between work and personal life, which has left me with little time to invest in my own learning.

A new hope

In late December / early January I took advantage of the $5 ebook deal on PacktPub.com, which is worth keeping an eye out for, and found a couple of books that tweaked my interest.

The 2 books covered Angular 5 with dotnet core and microservices. I have been playing around with dotnet core for a little while but couldn’t find a good guide that broke everything down to allow me to quickly learn Angular and put everything together, this ebook though was the solution and made everything click for me.

Learning experience

In order for me to gain a better understanding of how to build a project using Angular & dotnet core I have decided to take what I have learnt from the Angular & dotnet book and apply a microservice spin on it.

In my next set of posts I will walk you through how to setup a development environment and how to construct a project. This is just going to be putting the pieces together, not creating every bit of the application for you, for that I would advise you buy your own ebooks.

Moving on

Docker has also raised my interest and I look to dabble more in this now that I have purchased a couple of ebooks on this subject. I will include this as my knowledge and understanding grows.

SQL References for all Databases

So I’m working on migrating to a new instance of SQL and thought wouldn’t it be nice to see where all of my databases are referencing.

I had a quick look on Google, as you do, and found sys.sql_expression_dependencies, I thought this is great I can see all of the references for the current database.

After trying this out for a bit I then thought I should see if I could run this against each of my databases, this turned out to be easier than I thought by using sp_MSforeachdb, the only problem was that it returns multiple result sets not one long list.

After a bit of playing I came up with the following:

DECLARE @command varchar(1000)

DECLARE @RefInfo TABLE(
	[Database Name] NVARCHAR(255), 
	[Referencing Object] NVARCHAR(255), 
	[Referenced Database] NVARCHAR(255), 
	[Referenced Schema] NVARCHAR(255), 
	[Referenced Entity] NVARCHAR(255)
	)

SELECT @command = 'USE [?] 
SELECT  
	''?'' AS [Database Name], 
	OBJECT_NAME (referencing_id) AS referencing_object, 
	referenced_database_name, 
	referenced_schema_name, 
	referenced_entity_name 
FROM sys.sql_expression_dependencies 
WHERE referenced_database_name IS NOT NULL 
AND referenced_database_name NOT IN (
	''?'',
	''msdb'',
	''master''
	)
AND is_ambiguous = 0
AND ''?'' NOT IN(
	''master'',
	''msdb''
	)
' 

INSERT INTO @RefInfo EXEC sp_MSforeachdb @command

SELECT * FROM @RefInfo

This inserts all of the results into a single table variable, which is nice, and excludes any references to the master, msdb and the containing database.

Hope you get some use out of it.

Today was a sad day #!

Recently I have been using #! Crunchbang as my main OS, mostly because I use old equipment as anything from the last 7 years is suitable for most of my needs.

I was sad to hear that the sole developer of #! has decided to call it a day (http://crunchbang.org/forums/viewtopic.php?pid=416493#p416493). Good luck to him though.

I suppose I will have to decide on what I’m going to use going forward or whether I try to do my own thing.

Building a timer using jQuery

Ok, so another week and another post.

This week I will give you an example of how to create a timer using some of the methods from last weeks post.

The idea here is to use jQuery to build a timer, which is pretty much the same as the clock, however we will be adding some buttons to allow you to start/resume, pause & stop the timer.

The plan going forward is to use this as part of a time logging application, there are some parts of this code, such as the id parameter in the start function, which will become clearer as we move on.

First of all we will start by adding a new element to the body of our html document from last week, just stick it between the body tags:

<div class="timer">Timer</div>

The next thing we are going to do is add a script reference in the head of our document, just as we did last week, with a reference to a new script file:

<script src="timer.js"></script>

Lastly we need to create our script file, copy the code below and save it as timer.js:

$(document).ready(function(){
	
	//Build a point span element, so it can be re-used.
	var timerPoint = $('<span>').addClass("point").text(':');
	
	//Find the timer element
	var timerElem = $('.timer');
	//Clear the timer element
	timerElem.empty();
	//Build up the contents of the timer element using jQuery
	timerElem.append(
		$('<p>').addClass('time').append(
			$('<span>').addClass("hour")
		).append(
			timerPoint.clone()
		).append(
			$('<span>').addClass("minute")
		).append(
			timerPoint.clone()
		).append(
			$('<span>').addClass("second")
		).append(
			timerPoint.clone().text('.')
		).append(
			$('<span>').addClass("millisecond")
		)
		);
	
	//Create an object to contain information for the timer
	var timerObj = [];
	timerObj.currentDuration = 0;
	timerObj.originalStart = null;
	timerObj.currentStart = null;
	timerObj.currentId = null;
	timerObj.state = 0;
	
	//Create an anonymous function which will handle updating the text of the provided elements
	//target is the element we want to update
	//value is the numeric value we want to use
	//padding is a pattern we want to use to pad the value
	updateValue = (function(target, value, padding){

		value = padding + value.toString();
		value = value.substr(value.length - padding.length);
		
		if(target.text()!=value){
			
			target.text(value);//.hide().fadeIn('slow');
		}
	});
	
	//Add a function to the timer which will handle the start button event
	timerObj.start = function(id){
			
			this.currentId = id;
			
			if (timerObj.state == 0){
				timerObj.originalStart = new Date();
				timerObj.currentStart = timerObj.originalStart;
				timerObj.currentDuration = 0;
				timerObj.state = 1;
			} else {
				timerObj.currentStart = new Date();
			}
			
			timerObj.tick = function(){return setInterval(
				function(){
					var now = new Date();
					var duration = new Date((now - timerObj.currentStart)+ timerObj.currentDuration);
					
					updateValue($('.hour',timerElem),duration.getHours(),'00');
					updateValue($('.minute',timerElem),duration.getMinutes(),'00');
					updateValue($('.second',timerElem),duration.getSeconds(),'00');
					updateValue($('.millisecond',timerElem),duration.getMilliseconds(),'000');
					$('.point',timerElem).fadeOut(500).fadeIn(400);
				}
			,100);}();
			
		};
	
	//Add a function which will handle the pause button event
	timerObj.pause = function(){
			timerObj.state = 2;
			var now = new Date();
			timerObj.currentDuration = ((now.valueOf() - timerObj.currentStart.valueOf()) + timerObj.currentDuration.valueOf());
			clearInterval(timerObj.tick);
		};
	
	//Add a function which will handle the stop button event
	timerObj.stop = function(){
			timerObj.state = 0;
			var now = new Date();
			timerObj.currentDuration = ((now - timerObj.currentStart)+ timerObj.currentDuration);
			clearInterval(timerObj.tick);
		};
	
	//Add the buttons to the timer
	timerElem.append(
		$('<p>').addClass('controls').append(
			$('<button/>',{text: 'Start', click: function(){timerObj.start();}})
		).append(
			$('<button/>',{text: 'Pause', click: function(){timerObj.pause();}})
		).append(
			$('<button/>',{text: 'Stop', click: function(){timerObj.stop();}})
		)
		);
	
});

Building a simple clock in jQuery

Here is a quick example of how to build up an element using jQuery. I’m going to presume you know a bit about HTML and JavaScript, so i’m not going to bore you with all of that. You can always visit w3schools.com, which is a good reference for HTML, JavaScript, CSS …

To start off we create a very simple HTML document, copy the code below and save it as index.html:

<!doctype html>
<html lang="en">
	<head>
		<meta charset="utf-8">
		<title>Clock Demo</title>
		<script src="http://code.jquery.com/jquery-1.10.2.js"></script><!-- This is a link to a hosted jQuery library -->
		<script src="clock.js"></script><!-- This is our JavaScript file -->
	</head>
	<body>
		<!-- This div is the one we are going to use to hold our clock -->
		<div class="clock">Clock</div>
	</body>
</html>

Notice the two <script> tags, one references a hosted version of the jQuery library we are going to use, the other references an external file which we will use to create our clock.

Also notice the <div> element with a class of clock, we will use this in our clock.js JavaScript file to change the element into our clock.

Now we will create our JavaScript file, copy the code below and save it as clock.js:

//When the document is ready call this anonymous function
$(document).ready(function(){

	//Find the clock element
	var clockElem = $('.clock');
	
	//Write the content of the element directly
	//clockElem.html('<div class="time"><span class="hour"/><span class="point">:</span><span class="minute"/><span class="point">:</span><span class="second"/></div>');
	
	//Build the content of the element using jQuery
	//Empty the clock element.
	clockElem.empty();
	//Build a point span element, so it can be re-used.
	var clockPoint = $('<span>').addClass("point").text(':');
	//Append all of the required elements to the clock element.
	clockElem.append(
			$('<span>').addClass("hour")
		).append(
			clockPoint.clone()
		).append(
			$('<span>').addClass("minute")
		).append(
			clockPoint.clone()
		).append(
			$('<span>').addClass("second")
		);
	
	//Function object, this is used to hold one or more functions to be used within this function.
	var clockFunc = [];
	//A function to pad a value with additional 0 if required, and update the target if the value is different
	clockFunc.updateValue = (function(target, value){
		if(value < 10) {
			value = '0' + value;
		}
		if(target.text()!=value){	
			target.text(value).hide().fadeIn('slow');
		}
	});

	//We return a function which is repeated at a 1 second interval
	return setInterval(
		function(){
			var now = new Date(); //Get the current date & time
			clockFunc.updateValue($('.hour',clockElem),now.getHours()); //Update the hour value
			clockFunc.updateValue($('.minute',clockElem),now.getMinutes()); //Update the minute value
			clockFunc.updateValue($('.second',clockElem),now.getSeconds()); //Update the second value
			$('.point',clockElem).hide().fadeIn('fast'); //Flash the separators
		}	
	,1000);
	
});

What we are doing in this file is calling an anonymous (unnamed) function when the document is in a ready state (i.e. the document structure is loaded). The function locates our <div> using the clock class.

We then build the content of the <div> element, either directly with HTML or by building the content.

After we have built the structure of the clock we then need to update it every second. To accomplish this we create another anonymous function which updates the contents of the clock. To do this we create a helper function updateValue that we can re-use to format the values, I have added it to an object but you could just declare it as a variable without the need to create the containing object.

I hope this makes at least a bit of sense, this post is a bit of a rush job so I will probably come back and edit it over the next day or so.

2015

In 2014 I only published 1 post, which to be honest is a lot less than I intended. I have written other draft posts but never got round to finishing them. I don’t do this full-time, only as a way to record things that I have done. Although, I am doing more than one bit of work in a whole year!

This year, 2015, my intent to publish a few series of posts covering the stuff I am interested in and trying to explain them. Hopefully this will help other people, it will help me to solidify things in my own mind as I can’t explain to others that which I don’t understand myself.

Some of the topics I am considering on covering are:

• Building an HTML5 application (HTML, CSS, JavaScript/jQuery, PHP)
• Building a C# application
• Building an Android application
• Working with Linux, coming from a Windows background

My plan is to publish at least 1 post a week, I will try to keep them short and concise but provide enough information to be helpful to those who need it. I will also be going back over the things I have done over the last year or so and post any content that I deem useful, so keep an eye out for that as well.

That’s the first post of the year done, time to start writing the other 51+.

CRM 4.0 – User unable to access Filtered views

This is one that stumped me for a little while.

I was trying to figure out why one of my users was unable to run an Excel macro which read information from our CRM database. The user has a CRM account with all of the nessesary permissions but was unable to view any information from any Filtered views.

I figured out that it wasn’t an issue with the Roles in CRM, or the Security group used by CRM, or the security on the views themselves but actually it was an entry in the dbo.SystemUserBase table in the CRM database.

The user used to have an apostrophe (‘ “single quote”) in their username in AD when they first joined, this was subsequently removed in their AD account but in the dbo.SystemUserBase table the apostrophe was still there.

When a Filtered View is queried it checks the dbo.SystemUserBase table and tries to match it to the current users domain name, provided by the SUSER_NAME() function. If the values do not match then no records will be returned in the view.

The fix is to edit the value in the DomainName field in the dbo.SystemUserBase table to reflect the users domain name in AD.

VLC transcode script

Did you know that if you have ever wanted to transcode a load of video in one go and wanted to do it using the command line you can use VLC.

To set it up complete the following steps:
Download the Zip version of VLC.

Setup the folder structure, run the following from the command window whilst in the root folder.

mkdir in
mkdir processed
mkdir out
mkdir vlc

Extract the contents of the downloaded Zip file into the VLC folder.

Copy and paste the following code into notepad and save as transcode.bat in the root folder:

@ECHO off
FOR %%a IN (in\*.mkv) DO (
	ECHO Transcoding "%%a" to "out\%%~na.mp4"
	"vlc\vlc.exe" -I dummy -vvv "%%a" --sout=#transcode{vcodec=h264,acodec=mpga,ab=128,channels=2}:standard{access=file,mux=mp4,dst="out\%%~na.mp4"} --file-caching=300 vlc://quit
	ECHO Moving "%%a" to "processed\%%~nxa"
	MOVE "%%a" "processed\%%~nxa"
	ECHO ----------------------------------------
	)
PAUSE

Drop your files, in the case above *.mkv, into the in folder and then double click on the transcode.bat file to transcode them.

To find out how to transcode to different formats have a look at the command-line help on the VideoLAN website.