Submitting final project

Project Proposal.md

@@ -0,0 +1,21 @@
+# Drawing with Fingers
+The main idea of our project is to have an interface where someone can use a webcam to trace their finger and use their finger to paint on the screen. We will do a lot of image processing and finger tracking work with OpenCV.
+
+## _Minimal Viable Product_:
+* A display that can trace finger movement in black on a white drawing canvas
+## _Stretch Goal_: 
+* Having a way to turn off the paint so it’s not continuous
+* Having two paint brushes/brushes whose hue can be tuned in real time
+
+## Learning Goals for each person:
+Tommy: I want to get better at programming with another person, and also with making sure that I use Git properly.
+Josh: My goal is to get to use OpenCV more which I have evaded ever since the beginning of last semester.
+
+## Possible Libraries to Use:
+The libraries that we will probably be using are OpenCV and Pygame. OpenCV will be how we can utilize the webcams on laptops to keep track of finger movements in the video frame. Pygame will be used to creative an interactive drawing canvas that will put down colors based on the finger movements. The saturation of the colors will stay constant. Different finger speed will then result in different values. We are hoping to incorporate the other hand/fingers in an interesting way so that we can adjust the hue of the brushes.
+
+## Goal before the mid-project check-in
+By mid-project check in, we plan on having a skeleton of our code decided, and have a plan for how to implement tracking the finger. We also plan on knowing if pygame and OpenCV have all the resources we need for this project.
+
+## Risks???
+The biggest risks might include if we don’t plan out our implementation well and if we have a lot of difficulty with classes. 

ProjectReflection.md

@@ -0,0 +1,49 @@
+# Interactive Programming Project Reflection
+
+This is the base repo for the interactive programming project for Software Design, Spring 2018 at Olin College.
+
+You need to install OpenCV for this project. Run this line to install: `$ pip install opencv-python`
+
+#### To run the program
+To run the program, please either download zip or type in `git clone https://github.com/Tweir195/InteractiveProgramming.git` in the command line. To run the program, please type `python main.py` in the command line.
+
+Our program allows for three different modes, which you can specify through the few additional commands in the command line:
+
+`-t`: The trailing mode where the line you control is of fixed length, so older points after a certain number of new points are drawn will disappear. This is the default mode in which the program will run.
+
+`-g`: The gaming mode in which you try to hit colored boxes and get scores and added length to your line upon hittng the boxes.
+
+`-l` or `--length`: An integer following this command will specify the starting length of the line in the trailing mode and the gaming mode. The default starting length is 3.
+
+`-d` or `--draw`: The drawing mode where the line you control will stay on the screen as long as you haven't quit.
+
+When the program is running, there are several keyboard interactions that you can use: Key `q` would allow you to quit the program; key `s` would allow you to save the current canvas; key `t` would allow you to toggle trailing ON or OFF.
+
+## Project Overview
+In this interactive programming project, we created a program that uses OpenCV to track a red object and makes up a drawing app by drawing in a canvas where the red object has been. There are a few modes that you can choose from: a game, a drawing canvas, and a canvas with a disappearing line.
+
+## Results
+The most basic function of our program is to draw a tailing line. Our program would first filter out the color red in the whole camera frame. After identifying the contour in the filtered red frame, the program records the coordinate of the center of contours as a point to draw lines with. In the end, we would have a list of such points. By drawing a line between every two consecutive points in the list, we would end up with the trajectory of the red object in the frame. The way we maintain the list is by appending new points to the list until the designated length has been reached and then just shifting out the first element in the list and appending the new point to the list.
+
+![alt text](https://github.com/Tweir195/InteractiveProgramming/blob/master/drawing.jpg)
+
+At the same time, there's the option to make all the points stay on the canvas. To implement this, the only slight variation is that we would keep appending the coordinates to the list of all the points instead of shifting out the first one every time.
+
+![alt text](https://github.com/Tweir195/InteractiveProgramming/blob/master/gaming.jpg)
+
+The gaming mode is just a step up from the simple trailing mode. When the gaming mode is enabled, the canvas class would place a randomly colored rectangle at a random location inside canvas. Upon a point falling inside the drawn rectangle, users will be given ten points, and the tail of the line will be made longer. The current rectangle disappears while a new one shows up at another randomly chosen location. At the start of the program, the current time of the program is also recorded. When a countdown from 30 finishes, the program will freeze gaming and display the scores that the user receives in this round.Whenever one second elapses, the countdown in the game mode will decrease by one.
+
+## Implementation: still needs a UML
+Our program includes two classes: `canvas` and `fingerTrack`.  The `canvas` class creates a canvas to be drawn on. The initialization of `canvas` involves three input parameters: the width of the canvas, the height, and the scaler to enlarge the canvas by. The width and the height are based off the size of the camera frame. The scaler is always 2 except in the gaming mode where a scaler of 1 is applied to allow for better accuracy when playing the game. The `fingerTrack` class performs color tracking and actual drawing on the canvas. Each location of the points to be drawn are also scaled with the scaler in the `canvas` class. It also takes the `canvas` class as an input parameter when drawing new points and lines.
+
+One of our most important data structures is a list. We use lists to store such data as the coordinates of the center of the contours and the colors we are using for the lines between two consecutive points. We chose lists because they are mutable and have built-in operations like `append`. As a result, instead of using a data type to store a large number of points that have existed in the past, we used lists so that every time a point comes in, we can simply shift out the first one and append the new element to the end of the list, thus making a big save on memory.
+
+In our beginning planning of the project, after we had decided on using OpenCV to make a drawing platform, we had a few options. We thought about trying to make a game similar to the snake game, but the snake was controlled by the web-cam on the computer. We also thought about just making a canvas on the screen that could be drawn on by the web-cam, and have the color be controlled by the speed of movement. We were unsure about how easily we would be able to make the snake game, and decided on making the drawing canvas. However, we finished this with plenty of time, and expanded into a game in addition to the drawing canvas.
+
+UML diagram:
+![alt text](https://github.com/Tweir195/InteractiveProgramming/blob/master/UML_Project_4.jpg)
+
+## Reflection
+One thing that would have helped us a lot was having a more appropriately scoped project that takes better advantage of classes. We ended up doing a cool project, but because we started in an easier project, we had to keep finding ways to expand. This made us experiment with the code, without realizing we should have made a separate class for each mode. We could improve it by optimizing it so that there is less lag time and also so that it is better at picking up only the item that we want (this will avoid the jagged lines that can happen when there is interference).
+
+One of the good things about the experience is that we used both pair programming and divide-and-conquer in the process. Our approach in dividing and conquer was based on the next tasks that need to get done instead of which files or classes specifically so that both us have a good understanding of the program as a whole. We were able to move along in the project quite steadily, not feeling stressed about not finishing our project.

README.md

@@ -1,2 +1,25 @@
-# InteractiveProgramming
+### Our initial project proposal can be found [here](https://github.com/QingmuDeng/InteractiveProgramming/blob/master/Project%20Proposal.md).
+### Our project reflection can be found [here](https://github.com/Tweir195/InteractiveProgramming/blob/master/ProjectReflection.md).
+
+You need to install OpenCV for this project. Run this line to install: `$ pip install opencv-python`
+
+# Interactive Programming
+
 This is the base repo for the interactive programming project for Software Design, Spring 2018 at Olin College.
+
+You need to install OpenCV for this project. Run this line to install: `$ pip install opencv-python`
+
+#### To run the program
+To run the program, please either download zip or type in `git clone https://github.com/Tweir195/InteractiveProgramming.git` in the command line. To run the program, please type `python main.py` in the command line.
+
+Our program allows for three different modes, which you can specify through the few additional commands in the command line:
+
+`-t`: The trailing mode where the line you control is of fixed length, so older points after a certain number of new points are drawn will disappear. This is the default mode in which the program will run.
+
+`-g`: The gaming mode in which you try to hit colored boxes and get scores and added length to your line upon hittng the boxes.
+
+`-l` or `--length`: An integer following this command will specify the starting length of the line in the trailing mode and the gaming mode. The default starting length is 3.
+
+`-d` or `--draw`: The drawing mode where the line you control will stay on the screen as long as you haven't quit.
+
+When the program is running, there are several keyboard interactions that you can use: Key `q` would allow you to quit the program; key `s` would allow you to save the current canvas; key `t` would allow you to toggle trailing ON or OFF.

canvas.py

@@ -0,0 +1,98 @@
+import numpy as np
+import cv2
+import random
+
+
+class canvas():
+
+    def __init__(self, width, height, scaler):
+        """Initizes a canvas class with following attributes
+
+        width: the width of the drawing canvas given in pixels
+        height: the height of the drawing canvas given in pixels
+        new_canvas: a numpy array of zeros with depth of 3
+        randx: a range of x pixel value to choose from
+        randy: a range of y pixel value to choose from
+        colorlist: white,red, green, blue, yellow, purple, orange to randomly choose from
+        boxsize: the size of the box appearing in the gaming mode
+        points: the score a user has earned
+        value: the score given to a user upon hitting one rectangle
+        run: a boolean that makes sure boxes appear and disappear accordingly
+        """
+        self.screen_scaler = scaler
+        self.width = int(width) * self.screen_scaler
+        self.height = int(height) * self.screen_scaler
+        self.new_canvas = np.zeros((self.height, self.width, 3), np.uint8)
+        self.randx = np.linspace(10,580*self.screen_scaler)
+        self.randy = np.linspace(10,380*self.screen_scaler)
+        self.colorlist = [(255,255,255), (0,0,255), (0,255,0), (255,0,0), (0,255,255), (255,0,188), (0,15,255)]
+        self.boxsize = 30
+        self.points = 0
+        self.value = 10
+        self.run = False
+
+    def set_color(self, B, G, R):
+        """Stores the BGR value
+        """
+        self.color = (B, G, R)
+
+    def set_bgColor(self):
+        """Applies the BGR value to the drawing canvas
+        """
+        self.new_canvas[:, :] = self.color
+
+    def show_canvas(self):
+        """Displays a canvas on the screen
+        """
+        cv2.imshow('newCanvas', self.new_canvas)
+
+    def save_drawing(self):
+        """This function allows users to save their drawing with a name of
+        their choice.
+        """
+        file_name = input('Please name your drawing: ')
+        cv2.imwrite(file_name+'.jpg', self.new_canvas)
+
+    def clear(self):
+        """This function clears the screen.
+        """
+        canvas.new_canvas = np.zeros((self.height, self.width, 3), np.uint8)
+
+    def make_rect(self):
+        self.xpos = int(random.choice(self.randx))
+        self.ypos = int(random.choice(self.randy))
+        self.color = random.choice(self.colorlist)
+
+    def show_rect(self):
+        """Draws a rectangle in the gaming mode.
+        """
+        cv2.rectangle(self.new_canvas, (self.xpos, self.ypos), (self.xpos+self.boxsize,self.ypos+self.boxsize), self.color, 3)
+        self.run = True
+
+    def in_rect(self, pointx, pointy):
+        """Decides whether a point is within a rectangle
+
+        pointx: x pixel position of the point
+        pointy: y pixel position of the point
+        """
+        if self.xpos<pointx<self.xpos+self.boxsize and self.ypos<pointy<self.ypos+self.boxsize:
+            self.run = False
+            self.make_rect()
+            return True
+
+    def addpoints(self, track):
+        self.points += self.value
+        track.pathlength += 1
+
+if __name__ == "__main__":
+    canvas1 = canvas(1280, 960)
+    canvas1.set_color(0, 0, 0)
+    canvas1.set_bgColor()
+    while True:
+        canvas1.show_canvas()
+        if cv2.waitKey(1) & 0xFF == ord('s'):
+            canvas1.save_drawing()
+            break
+        elif cv2.waitKey(1) & 0xFF == ord('q'):
+            break
+    cv2.destroyAllWindows()

fingerTrack.py

@@ -0,0 +1,138 @@
+import numpy as np
+import cv2
+import math
+
+
+class finger_track():
+
+    def __init__(self):
+        """Initiates the finger tracking class
+        """
+        self.frame_num = 0
+        self.cx = 0
+        self.cy = 0
+        self.path = []
+        self.clearpath = []
+        self.pathlength = 10
+        self.red_maskL = [np.array([0, 150, 100]), np.array([178, 150, 100])]
+        self.red_maskH = [np.array([1, 255, 255]), np.array([180, 255, 255])]
+        self.refreshDelay = 0
+        self.colors = []
+        self.dist = []
+
+    def map(self, x, oldL, oldH, newL, newH):
+        """This function maps a value from one range to a differnet range
+
+        x: the value in the old range
+        oldL: the lower limit of the old range of values
+        oldH: the upper limit of the old range of values
+        newL: the lower limit of the new range of values
+        newH: the upper limit of the new range of values
+
+        doctest:
+        >>> map(1, 0, 10, 0, 200)
+        20
+        >>> map(50, 0, 150, 0, 255)
+        85
+        """
+        return int(((x - oldL)/(oldH-oldL))*(newH-newL)+newL)
+
+    def brush_color(self, hue):
+        """This function takes in a uint8 value for hue and generate
+        a BGR color range """
+        color = np.uint8([[[hue, 255, 255]]])
+        return cv2.cvtColor(color, cv2.COLOR_HSV2BGR)
+
+    def BGR2HSV(self, frame):
+        """This functions takes in a frame and converts it from BGR
+        to HSV values
+        """
+        return cv2.cvtColor(frame, cv2.COLOR_BGR2HSV)
+
+    def red_mask(self, frame):
+        """This function generates a red mask based on the frame being
+        passed in
+        """
+        mask1 = cv2.inRange(frame, self.red_maskL[0], self.red_maskH[0])
+        mask2 = cv2.inRange(frame, self.red_maskL[1], self.red_maskH[1])
+        return (mask1 | mask2)
+
+    def shift(self, myList, myElement):
+        """Shift out the first element in this list and append the new element to the end
+
+        >>> shift([1, 2, 3], 5)
+        [2, 3, 5]
+        """
+        return myList[1:] + [myElement]
+
+    def find_center(self, mask, target, canvas, disappr=True):
+        """This function takes in a cv2 mask, find the center of the
+        contours in the mask, and draw a green dot at the center location
+        on the target frame
+        """
+        im2, contours, hierarchy = cv2.findContours(mask, 1, 2)
+        try:
+            if self.frame_num > self.refreshDelay or self.frame_num == 0:
+                cnt = contours[0]
+                M = cv2.moments(cnt)
+                #print(M['m10'] / M['m00'])
+                self.cx = int(M['m10'] / M['m00']) * canvas.screen_scaler
+                self.cy = int(M['m01'] / M['m00']) * canvas.screen_scaler
+                self.frame_num = 0
+                if len(self.path) <= 1:
+                    self.path.append((self.cx, self.cy))
+                elif len(self.path) >= 2:
+                    # Calculate the distance between the two newest point.
+                    pair = self.path[-1]
+                    diffx = abs(self.cx-pair[0])
+                    diffy = abs(self.cy-pair[1])
+                    distance = math.sqrt(diffx**2+diffy**2)
+                    if distance < 150:
+                        # print('Far enough')
+                        if len(self.path) < self.pathlength:
+                            self.path.append((self.cx, self.cy))
+                        else:
+                            if disappr:
+                                self.path = self.shift(self.path, (self.cx, self.cy))
+                            else:
+                                self.path.append((self.cx, self.cy))
+                        dist2hue = self.map(distance, 0.0, 150.0, 0.0, 255.0)
+                        paintColor = self.brush_color(dist2hue)
+                        if len(self.colors) < self.pathlength:
+                            self.colors.append((int(paintColor[0][0][0]), int(paintColor[0][0][1]), int(paintColor[0][0][2])))
+                        else:
+                            if disappr:
+                                self.colors = self.shift(self.colors, (int(paintColor[0][0][0]), int(paintColor[0][0][1]), int(paintColor[0][0][2])))
+                            else:
+                                self.colors.append((int(paintColor[0][0][0]), int(paintColor[0][0][1]), int(paintColor[0][0][2])))
+                        # print(self.colors)
+            cv2.circle(target, (self.cx, self.cy), 2, (0, 255, 0), -1)
+            self.notFound = False
+        except IndexError:
+            """"""
+            self.notFound = True
+
+    def draw(self, canvas):
+        """This function draws the lines on the canvas of the screen.
+        The default is that only the 20 newest points will be drawn on screen.
+        """
+        canvas.new_canvas = np.zeros((canvas.height, canvas.width, 3), np.uint8)
+        for i in range(len(self.path)):
+            if len(self.path) <= 1:
+                break
+
+            else:
+                if i < len(self.path)-2:
+                    cv2.line(canvas.new_canvas, self.path[i], self.path[i+1], self.colors[i], 3)
+
+        if len(self.path) > 4:
+            def det(p1, p2, p3, p4):
+                deltaA = p1[0] - p2[0]
+                deltaB = p1[1] - p2[1]
+                deltaC = p3[0] - p4[0]
+                deltaD = p3[1] - p4[1]
+                return deltaA * deltaD - deltaB * deltaC
+            div = det(self.path[-1], self.path[-2], self.path[-3], self.path[-4])
+            if div != 0 and not self.notFound:
+                """"""
+                # print('the two line intersects!!!')