⚠️ This writing is a work in progress.⚠️

Please read everything found on the mainpage before continuing; disclaimer and all.

About this Tutorial:

Note: There is a lot of quoting and paraphrasing on this page in particular. Be sure to check the links you want to learn more.

We can update it in-drive and then rerun the results in near-real-time.

Several post-render functions will be used to reduce network requests.

I intend to hook this up to our webpack scripts for all the benefits.

Config Env

%%capture
!pip install marko
!python -m pip install nbdev

from IPython.core.interactiveshell import InteractiveShell

InteractiveShell.ast_node_interactivity = "all"

FNS

These scripts will convert an nb to an html doc. NBdev and Marko among other libraries are used.

Misc Notes

Unsorted notes from previous nb

"nb.keys()"

'nb.keys()'

# cell_type: 'code', 
# metadata<{colab{base_uri, height}, outputid}>, 
# outputs[
#  { name"stdout", output_type"stream", text"FutureWarning: Method `add_children` is deprecated"},
#  {, output_type"execute_result", 'data': {'text/html': 'thisistheactualmap!','text/plain': '<folium.folium.Map at 0x7f6c0106ff60>'}, execution_count, metadata{tags}
# ]
#], source"#hide_input\n#collapse_output\n makefoliummap(data)"   }

display_data = ['text/html', 'text/plain']
stream
execute_result = ['image/jpeg', 'text/plain']
error

%html

{'output_type': 'display_data', 'data': {'text/html': <script console.log('hi')\n script>", 'text/plain': '<IPython.core.display.HTML object>'} }

raw markdown

{'output_type': 'stream', 'text': 'Thisis text \n'}

display( airbnb.head() ); Image(filename='test.jpg')

{'output_type': 'display_data', 'data': {'text/html': div>oooAKKKKAP/Z</div', 'text/plain': '<IPython.core.display.Image object>'}

Convert FN's

This next bit will run the conversion on a single file.

import IPython
import requests
from google.colab import output
t= """
outp = runit() 
v = 0 if not 'v' in globals() else v+1
# page = f"index{v}.html"
page = f"index.html"
print(page)
f = open(page, "a")
f.write( ''.join( [''.join(c) for c in outp] ) )
f.close()

# posted = requests.get( 'https://charleskarpati.com/version.php', params=[('v', v)], ).json()

# IPython.display.HTML( outp )
"""

Serve Files with Ngrok

You can test on the Colab by using a flask server which can be a bit tricky.

from flask_ngrok import run_with_ngrok
from flask import Flask
from numpy import random
import requests
app = Flask(__name__)
run_with_ngrok(app)   #starts ngrok when the app is run

def getHtml(filepath):
    with open(filepath, 'r') as file:
      data = file.read().replace('\n', '')
    return data
@app.route("/")
def index():  
  # Get version number using the localstorage. 
  # v = requests.get( 'https://charleskarpati.com/version.php', ).json()
  # location = f'index{v}.html'
  # print('\n \n LOCATION: ', location)
  # return getHtml(location)
  outp = runit('BniaScripts.ipynb')
  resp = ''.join( [''.join(c) for c in outp] )
  resp = resp.replace('<link rel="stylesheet" type="text/css" href="https://cdnjs.cloudflare.com/ajax/libs/twitter-bootstrap/3.3.7/css/bootstrap.min.css">', '')
  return resp

 
@app.route("/about")
def about():
    return getHtml('lines.html')
 
app.run()

Working with GDrive Directories, Github, NBDev

Connect to Drive

If you have content on your Google Drive, you can publish entire directorys.

project = 'dataplay' # 'datalabs' 'dataguide' 'datalabs' 'dataplay' 'VitalSigns', 'DevelopersDocumentation'
%cd ../{project}/

/content/drive/My Drive/Software Development Documents/dataplay

import requests 
from IPython.core.display import display, HTML

SETUP

addToHeader = """ <!-- ************************** MIDDLE --> <section class="middle row">"""
def addAfterNav(title): return f"""<div class="span8 content"> <!-- PAGE HEADER --> <div class="page-header"><h1><b>{title}</b><small></small></h1></div> <!-- end PAGE HEADER -->"""
addToFooter = """</div></section></div>"""

navTop = """
  <!-- ************************** SIDEBAR -->
  <aside class="span4 content sidebar">
    <!-- SEARCH - WIDGET -->
    <div class="search-widget clearfix">
      <span style="display: block;">Share </span><!-- Simple Share Buttons Adder (7.4.18) simplesharebuttons.com -->
      <div id="ssba-classic-2" class="ssba ssbp-wrap ssbp--theme-1">
        <div style="display: flex;">
          <a data-site="" class="ssba_facebook_share" href="https://www.facebook.com/sharer.php?u=http://bniajfi.org/data_day/" target="_blank">
            <img src="https://bniajfi.org/wp-content/plugins/simple-share-buttons-adder/buttons/somacro/facebook.png" style="padding-right: 20px; width: 35px;" title="Facebook" class="ssba ssba-img" alt="Share on Facebook">
            <div title="Facebook" class="ssbp-text" style="display: none;">Facebook</div>
          </a>
          <a data-site="" class="ssba_twitter_share" href="https://twitter.com/share?url=http://bniajfi.org/data_day/&amp;text=Baltimore%20Data%20Day%20%28Week%29%20" target="&quot;_blank&quot;">
            <img src="https://bniajfi.org/wp-content/plugins/simple-share-buttons-adder/buttons/somacro/twitter.png" style="padding-right: 20px; width: 35px;" title="Twitter" class="ssba ssba-img" alt="Tweet about this on Twitter">
            <div title="Twitter" class="ssbp-text" style="display: none;">Twitter</div>
          </a>
          <a data-site="linkedin" class="ssba_linkedin_share ssba_share_link" href="https://www.linkedin.com/shareArticle?mini=true&amp;url=http://bniajfi.org/data_day/" target="&quot;_blank&quot;">
            <img src="https://bniajfi.org/wp-content/plugins/simple-share-buttons-adder/buttons/somacro/linkedin.png" style="padding-right: 20px; width: 35px;" title="LinkedIn" class="ssba ssba-img" alt="Share on LinkedIn">
            <div title="Linkedin" class="ssbp-text" style="display: none;">Linkedin</div>
          </a>
          <a data-site="email" class="ssba_email_share" href="mailto:?subject=Baltimore%20Data%20Day%20(Week)&amp;body=%20http://bniajfi.org/data_day/">
            <img src="https://bniajfi.org/wp-content/plugins/simple-share-buttons-adder/buttons/somacro/email.png" style="width: 35px;" title="Email" class="ssba ssba-img" alt="Email this to someone">
            <div title="email" class="ssbp-text" style="display: none;">email</div>
          </a>
        </div>
      </div>
    </div>
    <!-- end SEARCH - WIDGET -->
    <!-- WIDGET -->
    <div class="widget clearfix" style="display: block; margin-top: 5px;">
      <h3>Data Day Events</h3>
      <p>There are no upcoming events at this time.</p>
    </div>
    <div class="widget clearfix" style="display: block; margin-top: 5px;">
      <h3>Site Navigation</h3>
      <div class="menu-newsnav-container">
        <ul id="menu-newsnav" class="menu">
          <li id="menu-item-1524" class="menu-item menu-item-type-post_type menu-item-object-page menu-item-home current-menu-ancestor current_page_ancestor menu-item-has-children menu-item-1524">
            <a href="https://bniajfi.org/">Home</a>
            <ul class="sub-menu">
              <li id="menu-item-1528" class="menu-item menu-item-type-post_type menu-item-object-page menu-item-has-children menu-item-1528">
                <a href="https://bniajfi.org/vital_signs/">Vital Signs 19</a>
                <ul class="sub-menu">
                  <li id="menu-item-1529" class="menu-item menu-item-type-post_type menu-item-object-page menu-item-1529"><a href="https://bniajfi.org/vital_signs/fullreport/">Vital Signs – Full Report</a></li>
                  <li id="menu-item-1532" class="menu-item menu-item-type-post_type menu-item-object-page menu-item-1532"><a href="https://bniajfi.org/vital_signs/cprofiles/">Community Profiles</a></li>
                  <li id="menu-item-1531" class="menu-item menu-item-type-post_type menu-item-object-page menu-item-1531"><a href="https://bniajfi.org/vital_signs/data_downloads/">Data Downloads</a></li>
                  <li id="menu-item-1530" class="menu-item menu-item-type-post_type menu-item-object-page menu-item-1530"><a href="https://bniajfi.org/vital_signs/archives/">Archives</a></li>
                </ul>
              </li>
  """

navMid = """
            <li id="menu-item-1526" class="menu-item menu-item-type-post_type menu-item-object-page current-menu-ancestor current-menu-parent current_page_parent current_page_ancestor menu-item-has-children menu-item-1526">
              <a href="https://bniajfi.org/news_events/">News &amp; Events</a>
              <ul class="sub-menu">
                <li id="menu-item-1527" class="menu-item menu-item-type-post_type menu-item-object-page current-menu-item page_item page-item-109 current_page_item menu-item-1527"><a href="https://github.com/bnia/">All Repositories </a></li>
                <li id="menu-item-1527" class="menu-item menu-item-type-post_type menu-item-object-page current-menu-item page_item page-item-109 current_page_item menu-item-1527"><a href="https://bniajfi.org/VitalSigns/">Vital Signs Scripts </a></li>
                <li id="menu-item-1527" class="menu-item menu-item-type-post_type menu-item-object-page current-menu-item page_item page-item-109 current_page_item menu-item-1527"><a href="https://github.com/dataplay/">Dataplay Scripts</a></li>
                <li id="menu-item-1527" class="menu-item menu-item-type-post_type menu-item-object-page current-menu-item page_item page-item-109 current_page_item menu-item-1527"><a href="https://github.com/dataguide/">Data Handling Guidebook</a></li>
                <li id="menu-item-1527" class="menu-item menu-item-type-post_type menu-item-object-page current-menu-item page_item page-item-109 current_page_item menu-item-1527"><a href="https://github.com/datalabs/">Data Laboratories</a></li>
              </ul>
            </li>
"""

if (project == 'projectDocumentation'):
  navMid = """
    <li id="menu-item-1526" class="menu-item menu-item-type-post_type menu-item-object-page current-menu-ancestor current-menu-parent current_page_parent current_page_ancestor menu-item-has-children menu-item-1526">
      <a href="https://bniajfi.org/news_events/">BNIA Scripts</a>
      <ul class="sub-menu">
        <li id="menu-item-1527" class="menu-item menu-item-type-post_type menu-item-object-page current-menu-item page_item page-item-109 current_page_item menu-item-1527"><a href="https://bniajfi.org/bnia_scripts/">Bnia Scripts </a></li>
        <li id="menu-item-1527" class="menu-item menu-item-type-post_type menu-item-object-page current-menu-item page_item page-item-109 current_page_item menu-item-1527"><a href="https://github.com/bniajfi/">Bnia Github</a></li>
      </ul>
    </li>
  """

navBottom = """
            </ul>
          </li>
        </ul>
      </div>
    </div>
    <!-- end WIDGET -->
    <!-- WIDGET -->
    <!-- end WIDGET -->
    <!-- WIDGET -->
    <!-- end WIDGET -->
  </aside>
  <!-- ************************** end SIDEBAR -->
  <!-- ************************** CONTENT -->
  <div class="span8 content">
  <!-- end WIDGET -->
  <!-- WIDGET -->
  <!-- end WIDGET -->
  <!-- WIDGET -->
  <!-- end WIDGET -->
  </aside>
  <!-- ************************** end SIDEBAR -->
    """

nav = navTop + navMid + navBottom

Run one

ls

build/           dist/  LICENSE   MANIFEST.in  README.md     setup.py
CONTRIBUTING.md  docs/  Makefile  notebooks/   settings.ini

%cd notebooks

/content/drive/My Drive/Software Development Documents/dataguide/notebooks

ls

01_Github.ipynb        07_Javascript_Packages.ipynb
02_Colabs.ipynb        08_Cloud_Functions.ipynb
03_Shell_Basics.ipynb  09_Browser_Extensions.ipynb
04_APIs.ipynb          images/
05_Webscraping.ipynb   index.ipynb
06_Nbdev.ipynb

from flask_ngrok import run_with_ngrok
from flask import Flask
from numpy import random
import requests

header = requests.get('https://bniajfi.org/wp-json/wp/v2/header')
header = header.content.decode('utf-8')[:-4]
header= header.replace('/wp-content/uploads/2021/02/bnia_text.png', 'https://bniajfi.org/wp-content/uploads/2021/02/bnia_text.png')
header = header + addToHeader + nav
footer = requests.get('https://bniajfi.org/wp-json/wp/v2/footer')
footer = addToFooter+footer.content.decode('utf-8')[:-4] 
app = Flask(__name__)
run_with_ngrok(app)   #starts ngrok when the app is run
o = """
def getHtml(filepath):
    with open(filepath, 'r') as file:
      data = file.read().replace('\n', '')
    return data
"""
@app.route("/")
def index():  
  def template(title, x): return f'{header}{title}{x}{footer}'
  def replace(txt): 
    txt = txt.replace('<link rel="stylesheet" type="text/css" href="https://cdnjs.cloudflare.com/ajax/libs/twitter-bootstrap/3.3.7/css/bootstrap.min.css">', '')
    return txt
  # Get version number using the localstorage. 
  # v = requests.get( 'https://charleskarpati.com/version.php', ).json()
  # location = f'index{v}.html'
  # print('\n \n LOCATION: ', location)
  # return getHtml(location)
  infilename = 'index.ipynb'
  outp = runit(infilename)
  title = "".join(re.findall('[a-zA-Z][^a-zA-Z]*', 'index.ipynb'[:-6]))
  resp = ''.join( [''.join(c) for c in outp] )
  resp = replace(resp)
  resp = template( addAfterNav(title) , resp)
  return resp
 
@app.route("/about")
def about():
    return getHtml('lines.html')
 
app.run()

Run all

project = 'VitalSigns' # 'dataguide' 'datalabs' 'dataplay' 'VitalSigns', 'DevelopersDocumentation'
%cd ../{project}/

/content/drive/My Drive/Software Development Documents/VitalSigns

ls

build/                 example.csv    Hhchpov.geojson    README.md
CONTRIBUTING.md        hhchpov15.jpg  LICENSE            settings.ini
CSA-to-Tract-2010.csv  hhchpov16.jpg  Makefile           setup.py
dataplay/              hhchpov17.jpg  MANIFEST.in        test.gif
dataplay.egg-info/     hhchpov18.jpg  notebooks/         tracts_data.csv
dist/                  hhchpov19.jpg  part5negative.png
docs/                  Hhchpov.csv    part5positive.png

!git rm -r docs -f

!rm docs -r

mkdir docs

ls

import os 

def template(title, x): return f'{header}{title}{x}{footer}'
def replace(txt): 
  txt = txt.replace('<link rel="stylesheet" type="text/css" href="https://cdnjs.cloudflare.com/ajax/libs/twitter-bootstrap/3.3.7/css/bootstrap.min.css">', '')
  return txt

# traverse whole directory
for file in os.listdir(r'notebooks'): 
  # check the extension of files 
  if (file.endswith('.ipynb') ): 
      # print whole path of files 
      print(os.path.join('notebooks', file)) 
      title = "".join(re.findall('[a-zA-Z][^a-zA-Z]*', file[:-6]))
      page = f"./docs/{title}.html"
      outp = runit(os.path.join('notebooks', file))
      resp = ''.join( [''.join(c) for c in outp] )
      resp = replace(resp)
      resp = template( addAfterNav(title) , resp)
      try: f = open(page, "a")
      except: f = open(page[1:], "a")
      f.write( resp )
      f.close()

      # send it to my websesrver for saving.

! nbdev_build_docs --fname '' --mk_readme TRUE

No notebooks were modified
converting /content/drive/My Drive/Software Development Documents/VitalSigns/notebooks/index.ipynb to README.md

Publishing a new Repo

project = 'datalabs' # 'dataguide' 'datalabs' 'dataplay' 'VitalSigns', 'DevelopersDocumentation'
%cd ../{project}/

/content/drive/My Drive/Software Development Documents/datalabs

ls

build/              dist/    Makefile     README.md
CONTRIBUTING.md     docs/    MANIFEST.in  settings.ini
datalabs.egg-info/  LICENSE  notebooks/   setup.py

! git init
! git config --global user.email "bniajfi@gmail.com"
! git config --global user.name "bniajfi"

Reinitialized existing Git repository in /content/drive/My Drive/Software Development Documents/datalabs/.git/

! git add *
! git commit -m "Second commit for styling"
! git branch -M main

The following paths are ignored by one of your .gitignore files:
build
datalabs.egg-info
dist
Use -f if you really want to add them.
On branch main
Untracked files:
	.gitignore

nothing added to commit but untracked files present

tokenVS = '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'
token = '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'
username = 'BNIA'
print(f'https://{token}@github.com/{username}/{project}.git')
! git remote remove origin 
! git remote add origin 'https://{token}@github.com/{username}/{project}.git'
#! git push -u origin main 
!git push 'https://{token}@github.com/{username}/{project}.git'

https://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@github.com/BNIA/datalabs.git
Counting objects: 15, done.
Delta compression using up to 2 threads.
Compressing objects: 100% (15/15), done.
Writing objects: 100% (15/15), 2.87 KiB | 1.44 MiB/s, done.
Total 15 (delta 13), reused 0 (delta 0)
remote: Resolving deltas: 100% (13/13), completed with 12 local objects.
To https://github.com/BNIA/datalabs.git
   8fc786b..cdfbe30  main -> main

Pypi

ls

build/           dataplay.egg-info/  LICENSE      notebooks/    setup.py
CONTRIBUTING.md  dist/               Makefile     README.md
dataplay/        docs/               MANIFEST.in  settings.ini

! pip install twine
! nbdev_bump_version
! make pypi

ls

build/           dataplay.egg-info/  LICENSE      notebooks/    setup.py
CONTRIBUTING.md  dist/               Makefile     README.md
dataplay/        docs/               MANIFEST.in  settings.ini

The “src refspec master does not match any” error occurs if you have forgotten to add the files you have changed to a commit and try to push those changes to a remote repository before you make the first commit in your repository.

'https://docs.github.com/en/github/importing-your-projects-to-github/importing-source-code-to-github/adding-an-existing-project-to-github-using-the-command-line'
'https://stackoverflow.com/questions/61424599/error-when-pushing-files-to-git-in-colab-fatal-could-not-read-username-for-ht'
'https://stackoverflow.com/questions/492558/removing-multiple-files-from-a-git-repo-that-have-already-been-deleted-from-disk'
'https://github.community/t/error-refname-refs-heads-master-not-found/144737/13'
'https://github.com/settings/tokens'
'https://github.com/settings/tokens'
'https://stackoverflow.com/questions/61424599/error-when-pushing-files-to-git-in-colab-fatal-could-not-read-username-for-ht'

headerPHPEndsLike = """
</li>
</ul>                        <!-- end PRIMARY NAV -->
                        
						
                    </nav>
			</div>
        </header>
		</div>
        <!-- ************************** end HEADER -->
	<!-- ************************** CONTAINER -->
    <div class="container">
	 
                       null
"""

headerDoesntHave = """
  <div class="container">
        <!-- ************************** MIDDLE -->
        <section class="middle row">
			<!-- ************************** SIDEBAR -->
					<aside class="span4 content sidebar">
          ...
          </aside>
          class="span8 content"
"""

footerDoesntHave = """
  </div> <- end content div
  </section> <- end section
  </div> <- end container
"""

footerPHPStartsLike = """    	<div class="row"  style="background-color: #FDFDFD; border-top: 15px solid #007db6; -moz-box-shadow: inset 0 0 5px #888; -webkit-box-shadow: inset 0 0 5px#888; box-shadow: inner 0 0 5px #888;">
		<div class="container" style="margin-top: 10px;">
		<style>#copy-text-lakin{overflow:hidden;}#copy-text-lakin::after{clear:both; content:' '; display:block}#copy-text-lakin>*{width:822px; float:left; margin:0}#copy-text-lakin>#copy-text-lak{margin-left:-822px; background-color:#007db6; width:100%}</style>
            <div class="row">
                <!-- widget -->
				<div class="span3 widget">
					<h3><b>SEARCH</b></h3><form method="get" action="http://bniajfi.org/">
"""

nb 2 html

About this Tutorial:

Config Env

FNS

`convertNb`[source]

`cleanCell`[source]

`processCode`[source]

`getFlags`[source]

`makeDetails`[source]

`processSource`[source]

`processOutput`[source]

`runit`[source]

Misc Notes

Convert FN's

Serve Files with Ngrok

Working with GDrive Directories, Github, NBDev

Connect to Drive

SETUP

Run one

Run all

Publishing a new Repo

Pypi

nb 2 html

About this Tutorial:

Config Env

FNS

convertNb[source]

cleanCell[source]

processCode[source]

getFlags[source]

makeDetails[source]

processSource[source]

processOutput[source]

runit[source]

Misc Notes

Convert FN's

Serve Files with Ngrok

Working with GDrive Directories, Github, NBDev

Connect to Drive

SETUP

Run one

Run all

Publishing a new Repo

Pypi

`convertNb`[source]

`cleanCell`[source]

`processCode`[source]

`getFlags`[source]

`makeDetails`[source]

`processSource`[source]

`processOutput`[source]

`runit`[source]